fivetran / dbt_jira

Data models for Fivetran's Jira connector built using dbt.

Jira Transformation dbt Package (Docs)

📣 What does this dbt package do?

• Produces modeled tables that leverage Jira data from Fivetran's connector in the format described by this ERD and builds off the output of our Jira source package.
• Enables you to better understand the workload, performance, and velocity of your team's work using Jira issues. It performs the following actions:
  • Creates a daily issue history table so you can quickly create agile reports, such as burndown charts, along any issue field.
  • Enriches the core issue table with relevant data regarding its workflow and current state.
  • Aggregates bandwidth and issue velocity metrics along projects and users.
• Generates a comprehensive data dictionary of your source and modeled Jira data through the dbt docs site.

The following table provides a detailed list of all models materialized within this package by default.

TIP: See more details about these models in the package's dbt docs site.

Model	Description
jira__daily_issue_field_history	Each record represents a day in which an issue remained open, enriched with data about the issue's sprint, its status, and the values of any fields specified by the issue_field_history_columns variable.
jira__issue_enhanced	Each record represents a Jira issue, enriched with data about its current assignee, reporter, sprint, epic, project, resolution, issue type, priority, and status. It also includes metrics reflecting assignments, sprint rollovers, and re-openings of the issue. Note that all epics are considered issues in Jira and are therefore included in this model (where issue_type='epic').
jira__project_enhanced	Each record represents a project, enriched with data about the users involved, how many issues have been opened or closed, the velocity of work, and the breadth of the project (i.e., its components and epics).
jira__user_enhanced	Each record represents a user, enriched with metrics regarding their open issues, completed issues, the projects they work on, and the velocity of their work.

🎯 How do I use the dbt package?

Step 1: Prerequisites

To use this dbt package, you must have the following:

• At least one Fivetran Jira connector syncing data into your destination.
• A BigQuery, Snowflake, Redshift, Databricks, or PostgreSQL destination.

Databricks Dispatch Configuration

If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your dbt_project.yml. This is required in order for the package to accurately search for macros within the dbt-labs/spark_utils then the dbt-labs/dbt_utils packages respectively.

dispatch:
  - macro_namespace: dbt_utils
    search_order: ['spark_utils', 'dbt_utils']

Database Incremental Strategies

Models in this package that are materialized incrementally are configured to work with the different strategies available to each supported warehouse.

For BigQuery and Databricks All Purpose Cluster runtime destinations, we have chosen insert_overwrite as the default strategy, which benefits from the partitioning capability.

For Databricks SQL Warehouse destinations, models are materialized as tables without support for incremental runs.

For Snowflake, Redshift, and Postgres databases, we have chosen delete+insert as the default strategy.

Regardless of strategy, we recommend that users periodically run a --full-refresh to ensure a high level of data quality.

Step 2: Install the package

Include the following jira package version in your packages.yml file:

TIP: Check dbt Hub for the latest installation instructions or read the dbt docs for more information on installing packages.

packages:
  - package: fivetran/jira
    version: [">=0.17.0", "<0.18.0"]

Step 3: Define database and schema variables

By default, this package runs using your destination and the jira schema. If this is not where your Jira data is (for example, if your Jira schema is named jira_fivetran), add the following configuration to your root dbt_project.yml file:

vars:
    jira_database: your_destination_name
    jira_schema: your_schema_name 

Step 4: Disable models for non-existent sources

Your Jira connector may not sync every table that this package expects. If you do not have the SPRINT, COMPONENT, or VERSION tables synced, add the respective variables to your root dbt_project.yml file. Additionally, if you want to remove comment aggregations from your jira__issue_enhanced model, add the jira_include_comments variable to your root dbt_project.yml:

vars:
    jira_using_sprints: false   # Disable if you do not have the sprint table or do not want sprint-related metrics reported
    jira_using_components: false # Disable if you do not have the component table or do not want component-related metrics reported
    jira_using_versions: false # Disable if you do not have the versions table or do not want versions-related metrics reported
    jira_using_priorities: false # disable if you are not using priorities in Jira
    jira_include_comments: false # This package aggregates issue comments so that you have a single view of all your comments in the jira__issue_enhanced table. This can cause limit errors if you have a large dataset. Disable to remove this functionality.

(Optional) Step 5: Additional configurations

Define daily issue field history columns

The jira__daily_issue_field_history model generates historical data for the columns specified by the issue_field_history_columns variable. By default, the only columns tracked are status, status_id, and sprint, but all fields found in the Jira FIELD table's field_name column can be included in this model. The most recent value of any tracked column is also captured in jira__issue_enhanced.

If you would like to change these columns, add the following configuration to your dbt_project.yml file. After adding the columns to your dbt_project.yml file, run the dbt run --full-refresh command to fully refresh any existing models:

IMPORTANT: If you wish to use a custom field, be sure to list the field_name and not the field_id. The corresponding field_name can be found in the stg_jira__field model.

vars:
    issue_field_history_columns: ['the', 'list', 'of', 'field', 'names']

Adjust the field-grain for issue field history transformations if duplicate field names

This package provides the option to use field_name instead of field_id as the field-grain for issue field history transformations. By default, the package strictly partitions and joins issue field data using field_id. However, this assumes that it is impossible to have fields with the same name in Jira. For instance, it is very easy to create another Sprint field, and different Jira users across your organization may choose the wrong or inconsistent version of the field. As such, the jira_field_grain variable may be adjusted to change the field-grain behavior of the issue field history models. You may adjust the variable using the following configuration in your root dbt_project.yml.

vars:
    jira_field_grain: 'field_name' # field_id by default

Extend the history of an issue past its closing date

This packages allows you the option to utilize a buffer variable to bring in issues past their date of close. This is because issues can be left unresolved past that date. This buffer variable ensures that this daily issue history will not cut off field updates to these particular issues.

You may adjust the variable using the following configuration in your root dbt_project.yml.

vars:
    jira_issue_history_buffer: insert_number_of_months # 1 by default

Change the build schema

By default, this package builds the Jira staging models within a schema titled (<target_schema> + _jira_source) and your Jira modeling models within a schema titled (<target_schema> + _jira) in your destination. If this is not where you would like your Jira data to be written to, add the following configuration to your root dbt_project.yml file:

models:
    jira_source:
      +schema: my_new_schema_name # leave blank for just the target_schema
    jira:
      +schema: my_new_schema_name # leave blank for just the target_schema

Change the source table references

If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:

IMPORTANT: See this project's dbt_project.yml variable declarations to see the expected names.

vars:
    jira_<default_source_table_name>_identifier: your_table_name 

Lookback Window

Records from the source can sometimes arrive late. Since several of the models in this package are incremental, by default we look back 3 days to ensure late arrivals are captured while avoiding the need for frequent full refreshes. While the frequency can be reduced, we still recommend running dbt --full-refresh periodically to maintain data quality of the models.

To change the default lookback window, add the following variable to your dbt_project.yml file:

vars:
  jira:
    lookback_window: number_of_days # default is 3

(Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™

Expand for details

Fivetran offers the ability for you to orchestrate your dbt project through Fivetran Transformations for dbt Core™. Learn how to set up your project for orchestration through Fivetran in our Transformations for dbt Core setup guides.

🔍 Does this package have dependencies?

This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the dbt hub site.

IMPORTANT: If you have any of these dependent packages in your own packages.yml file, we highly recommend that you remove them from your root packages.yml to avoid package version conflicts.

packages:
    - package: fivetran/jira_source
      version: [">=0.7.0", "<0.8.0"]

    - package: fivetran/fivetran_utils
      version: [">=0.4.0", "<0.5.0"]

    - package: dbt-labs/dbt_utils
      version: [">=1.0.0", "<2.0.0"]

    - package: dbt-labs/spark_utils
      version: [">=0.3.0", "<0.4.0"]

🙌 How is this package maintained and can I contribute?

Package Maintenance

The Fivetran team maintaining this package only maintains the latest version of the package. We highly recommend you stay consistent with the latest version of the package and refer to the CHANGELOG and release notes for more information on changes across versions.

Contributions

A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions!

We highly encourage and welcome contributions to this package. Check out this dbt Discourse article on the best workflow for contributing to a package!

🏪 Are there any resources available?

• If you have questions or want to reach out for help, please refer to the GitHub Issue section to find the right avenue of support for you.
• If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our Feedback Form.
• Have questions or want to be part of the community discourse? Create a post in the Fivetran community and our team along with the community can join in on the discussion!

dbt_jira's Issues

[Feature] Revise documentation for vars `jira_field_grain` and `issue_field_history_columns`

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Regardless whether I set jira_field_grain as 'field_id' or 'field_name', I don't seem to be able to use the value 'customfield_10028' to define the field in issue_field_history_columns. I can use the value 'Story Points' and it works fine with both grain settings.

Relevant error log or model output

No response

Expected behavior

I expect to be able to use the value 'customfield_10028' to define the fields in issue_field_history_columns

dbt Project configurations

Variables for the Fivetran JIRA package

jira_database: dw
jira_schema: jira
issue_field_history_columns: ['Story Points', 'customfield_10028']
jira_field_grain: 'field_id' # field_id or field_name

Package versions

packages:

• package: dbt-labs/codegen
  version: 0.12.1
• package: fivetran/ad_reporting
  version: 1.7.0
• package: fullstorydev/dbt_fullstory
  version: 0.5.1
• package: fivetran/jira
  version: 0.15.0

What database are you using dbt with?

snowflake

dbt Version

1.7

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Properly support multiple field selections in `daily_issue_field_history`

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

There will be times when a Jira user will assign multiple field values to a particular issue (i.e. multiple components or sprints). Currently, the jira__daily_issue_field_history model does bring in values, but only the ids associated with those values.

In the example below, we've added multiple components and sprints to a particular issue, but could only bring in the ids into the model.

We should be able to support bringing in all field name values into our models so our customers can more easily understand the history of issues and all the various fields associated with it, day over day.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

[Feature] <Add optional Project Configuration for specifying Jira Projects>

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

We use Jira for software engineering work and for operations work. I would like to use this package to analyze the software engineering tickets, but not the operations tickets. These tickets are separated by projects.

Describe alternatives you've considered

I could use dbt to create copies of the jira tables that only contain tickets in the software engineering project.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

[Bug] Assignee names, reporter names, emails etc not appearing how they should

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

This does not look right to me. Don't think this is a me issue but do say if it is

Relevant error log or model output

No response

Expected behavior

Would expect to see the information aligned to the column names

dbt Project configurations

Package versions

What database are you using dbt with?

snowflake

dbt Version

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Allow for Custom Field Names (not IDs) to be Present in Final Issue Model

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Currently, the dbt package allows for custom fields to be passed through to the final jira__issue_enhanced model. However, these are only the IDs if the fields are custom (ie. not sprint, story point, assignee, etc.).

This request is to leverage the field table and map the correct custom field name to the final jira_issue_enhanced and possibly jira__issue_field_history final models.

Describe alternatives you've considered

Attempting to do this in a model on top of the package. However, it is not as intuitive as we would like to join these field names into the final model fields. Therefore, it would be more ideal for the package to handle this operation.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

Any other Fivetran dbt_jira package users that have wished for this feature, please feel free to comment in the thread or like this feature request.

Thanks!

BUG - Duplicate in jira_issues

Are you a current Fivetran customer?
Philippine Marionnet, Data Analyst Product, Aircall

Describe the bug
There are some duplicates in the dbt table for jira_issues_custom_fields

Steps to reproduce

select issue_id, count(*)
from jira_issues_custom_fields 
group by 1;

Expected behavior
I would like to have a unique issue_id for each line.

Project variables configuration

A good package name should reflect your organization's name or the intended use of these models

name: 'aircall'
version: '1.0.0'
config-version: 2

This setting configures which "profile" dbt uses for this project.

profile: 'aircall'

source-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
data-paths: ["data"]
macro-paths: ["macros"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by dbt clean

• "target"

models:
aircall:
+materialized: view_if_not_exists
intermediate_marts:
+schema: intermediate
sources:
+schema: sources

vars:
dbt_date:time_zone: 'UTC'
phone_schema: "client_events_phone_staging"
android_schema: "client_events_android_staging"
ios_schema: "client_events_ios_staging"
browser_extension_schema: "client_events_browser_extension_staging"
dashboard_schema: "client_events_dashboard_staging"
dry_run : False #adding a flag variable to the compile step in order to avoid failure during compiling, when the current code depends on the data of a model which is not yet materialized
jira:
jira_include_comments: false
# ----- IF YOU WANT TO ADD A FIELD TO jira_changelog ----- #
field_columns: # lower case with _ if more than 1 word
- name: status
table: stg_jira__status
joining_id: status_id
source_field: status_name
- name: assignee
table: stg_jira__user
joining_id: user_id
source_field: user_display_name
- name: assignment_group
table: clean_jira_field_option
joining_id: field_id
source_field: field_name

# ----- IF YOU WANT TO ADD A FIELD TO jira_issues_d (and jira_issues) ----- #
issue_field_history_columns: ['customfield_10161', 'assignee', 'priority', 'customfield_10170','customfield_10207'] # variable used in jira__daily_issue_field_history

# ----- IF YOU WANT TO ADD A CUSTOM FIELD TO jira_issues ONLY (not to jira_issues_d)----- #
custom_fields_columns: ['product_category','severity','labels','blast_radius','region','service_impact','symptoms']

# ----- IF YOU WANT TO ADD A SPECIAL CUSTOM FIELD TO jira_issues ONLY (not to jira_issues_d)----- #
# ----- special custom field: field for which the value is not represented by an id but its value in int_jira__combine_field_histories----- #
special_custom_fields_columns: ['story_points', 'email_verification','issue_link_count']

intercom:
using_contact_tags: false
using_team: false

Package Version
packages:

• package: dbt-labs/redshift
  version: [">=0.5.0", "<0.6.0"]
• package: fivetran/jira
  version: [">=0.5.0", "<0.6.0"]
• package: fivetran/intercom
  version: [">=0.3.0", "<0.4.0"]
• package: calogica/dbt_expectations
  version: [">=0.4.0", "<0.5.0"]

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context

Screenshots

Please indicate the level of urgency
It is impacting my reporting of a critical dashboard.

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Bug] `dbt_project.yml` needs Components specified as a `issue_field_history_columns` vars if customer is using components

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

We recently introduced new logic into jira__daily_issue_field_history that accounts for pulling components data directly into our model to avoid any overlap with field_option. However, this is currently throwing an error because we did not add the necessary variable logic in dbt_project.yml for when it isn't present inside issue_field_history.

Adding

vars:
     jira:
      issue_field_history_columns:
          - "Components" 

is necessary.

However, we will want this logic to compile only if the customer is using the components table. We might recommend using {% if var('jira_using_components', True) %} above issue_field_history_columns since we wouldn't want to call it if components wasn't a part of the model. See examples in the newest version of jira__daily_issue_field_history.

Relevant error log or model output

00:11:01  Database Error in model jira__daily_issue_field_history (models/jira__daily_issue_field_history.sql)
00:11:01    Name components not found inside joined at [101:65]
00:11:01    compiled Code at target/run/jira/models/jira__daily_issue_field_history.sql

Expected behavior

The models should compile but it is not finding the components column inside daily_issue_field_history if it is not set as a field.

dbt Project configurations

vars:
     jira:
      issue_field_history_columns:
          - "Components" 

Package versions

jira 0.12.1

What database are you using dbt with?

bigquery

dbt Version

dbt=1.3.0

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Allow for both custom_field_id and name if desired for passthrough column

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Currently, the issue_field_history_columns variable allows for the field name to be passed through to the final models. However, as addressed within this dbt Slack thread it was understood that the name field is too mutable and changes too frequently. Additionally, there is no restrictions for Jira users to name two different fields the same 😱

As such, I want to consider a possibility for the package to allow the name of the custom fields to be passed through in addition to the id as well. The id is pretty hard to interpret for an end user (which is why we removed this feature before), but there is now a possibility for data integrity issues as the name is mutable. I would like to consider a solution where there are two variables.

• A variable for the name of the custom field to be passed through
• The id of the field to be passed through

This way the package will have the best of both worlds!

I would still like to keep the name version of the custom field as it is a lot easier for folks to decipher what the fields are when looking at their dbt_project.yml and for initial setup as well. However, I do believe we should include an option for folks to use id if they wanted to ensure the fields are immutable.

Describe alternatives you've considered

Currently there are no other alternatives other than the PR #58 I opened which is a working solution for this.

Please note that this PR is not the final version and just a working version of my thoughts on a solution.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

One thing we will need to communicate is that if a user leverages the id variable, the end model will still display the name. As such, they will still need to consider a --full-refresh if that field does in fact change.

Additionally, we should consider changing the name of the variable altogether. The name is no longer descriptive of what it does since it is used in more than just the issue_field_history end model. Just a thought.

README.md is an unformatted template

Toot toot
Beep beep

int_jira__issue_calendar_spine performs slowly on snowflake

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The model 'int_jira__issue_calendar_spine' is performing slowly on snowflake - taking about 10-15 minutes depending on the run.
Our instance has about 600M records for this table.
The query plans show that all records are being deleted/inserted for each run.
Should this be materialized as a table instead?

14:51:31 1455 of 1614 OK created sql incremental model AV_int_jira.int_jira__issue_calendar_spine [�[32mSUCCESS 650585743�[0m in 597.48s]
Shouldn't only a small % be loaded with incremental strategy not 99%?

Describe alternatives you've considered

Fork the repo and change myself ... but that requires upkeep :(

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Issue in jira__daily_issue_field_history when field changes several times a day

Hi team,

I noticed an issue in jira__daily_issue_field_history whenever a field changes several times a day.
In the exemple below, we see that everything is well recorded in int_jira__combine_field_histories. However, in int_jira__daily_field_history, where we are supposed to only look at the latest value of each field for each day and each issue, there are still two instances of the Assignee field for the same day... That messes up the data of all following models and eventually of the jira__daily_issue_field_history model.

Here's what I got after doing a full refresh of the models. The data is correct so I assume the issue comes from the incremental mode...

Do you think you could help me on this?
Thanks a lot,
Have a great day!
Marie

[Bug] Incremental runs cause null daily_history fields when issue is updated after being Closed for 30 days

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

When issues have had Status = Closed for 30+ days and are then updated, all columns other than date_day, issue_id, issue_day_id on the jira__daily_issue_field_history table become null. Since the most recent date for the issue receives all null values, the jira__issue_enhanced table is updated to all null values for that issue as well. Full-refresh-ing the package fixes this problem by creating a row for every date_day in between the updates.

Related Slack thread: https://getdbt.slack.com/archives/C01D1R2JLLA/p1681241427533479

Relevant error log or model output

Screenshot of problematic output included below.

Expected behavior

The daily table should record the new/current values of all fields on an issue regardless of how long it has been Closed for when a field is updated. The daily table does not need to include a row for every issue_date_day after an issue has been Closed for 30+ days, but if it is updated again, then it should record a new date_day row with the newly updated field values.

After the update, it does not need to record the next 30 days unless the issue was re-opened (but whatever you all think is best in this case works for me).

dbt Project configurations

version: '1.3.1'
config-version: 2

  jira:
    schema: dbt
  jira_source:
    schema: dbt_stg

  jira_database: raw
  jira_schema: jira 

  jira:
    issue_field_history_columns: [
      'Affects versions'
      ,'Approved By'
      ,'Assignee'
      ,'Battery Serial Number'
      ,...
      ,'Zip Number'
     ]

Package versions

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.0

  - package: fivetran/jira
    version: [">=0.13.0", "<0.14.0"]

What database are you using dbt with?

snowflake

dbt Version

Core:
  - installed: 1.3.1
  - latest:    1.5.0 - Update available!

Additional Context

Example of problematic output:

Corresponding issue's history in the Jira UI:

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Replicate Jira's Velocity Chart and Sprint Report metrics

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

We need a snapshot of story points that were "committed" at the beginning of a sprint and story points that were "completed" at the time the sprint ended. Teams would like to track sprint velocity over time in one place (ex: Tableau dashboard). Jira's interface for the Velocity Chart is limited to the past 7 Sprints. Jira's Client API does not currently provide these metrics.

Describe alternatives you've considered

Jira's Front-End API (greenhopper) provides the data needed but is unsupported by Jira, and access could be disabled anytime.

Jira Front-End API endpoints:

Velocity Chart:

• https://{site_name}.atlassian.net/rest/greenhopper/1.0/rapid/charts/velocity?rapidViewId={boardId}
• Returned data: sprint_id, estimated_points, completed_points

Sprint Report:

• https://{site_name}.atlassian.net/rest/greenhopper/1.0/rapid/charts/sprintreport?rapidViewId={boardId}&sprintId={sprintId}
• Returned data (below includes my renaming of the returned data):
  'id': 'issue_id',
  'key': 'issue_key',
  'typeName': 'issue_type',
  'priorityName': 'issue_priority',
  'done': 'issue_is_done',
  'assigneeName': 'issue_assignee',
  'epic': 'issue_epic_name',
  'epicField.issueId': 'issue_epic_id',
  'currentEstimateStatistic.statFieldValue.value': 'issue_EOS_points',
  'estimateStatistic.statFieldValue.value': 'issue_BOS_points',
  'status.id': 'issue_status_id',
  'status.name': 'issue_status_name',
  'status.statusCategory.id': 'issue_status_cat_id',
  'issue_type': 'issue_status_type',
  'added': 'issue_added_after_BOS',
  'completedIssuesEstimateSum': 'sprint_issue_points_completed_EOS'

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

Jira's Velocity Chart definitions

LISTAGG limit error on Redshift in int_jira__issues_comments

Hello !

As a Fivetran customer, I'm pretty excited to see there is some amazing work done on Jira, thank you very much for what you're doing :) !

I tried to run it on Redshift. However, int_jira__issues_comments is triggering an error :

  Result size exceeds LISTAGG limit
  DETAIL:  
    -----------------------------------------------
    error:  Result size exceeds LISTAGG limit
    code:      8001
    context:   LISTAGG limit: 65535
    query:     14079182
    location:  _rds_bin_padb.1.0.22169_data_exec_18_1965502738_f80d9adf9dfd682644c377b2083585066ccc41ef_0.cpp:644
    process:   query1_470_14079182 [pid=32338]

I think this is due to the fivetran_utils.string_agg. As I'm using Redshift, it's in fact using LISTAGG.
The problem is that in int_jira__issues_comments, comment.body (ie. a very large string) is a parameter of LISTAGG.

https://docs.aws.amazon.com/redshift/latest/dg/r_LISTAGG.html

"Returns
VARCHAR(MAX). If the result set is larger than the maximum VARCHAR size (64K – 1, or 65535), then LISTAGG returns the following error:"

Therefore, I don't think using LISTAGG for possible large strings is a good idea.

Could you fix it for Redshift users please? Thank you very much!

[Bug] Current Sprint and Sprint not handling nulls well

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

If an issue is removed from a sprint, it is still considering that last sprint (in history) as the current_sprint.

1. add issue to sprint
2. remove issue from sprint
3. run query:

select
  issue_key
  , sprint
from
  <db_name>.jira.issue_enhanced
where
  issue_key = <issue_key> 

Relevant error log or model output

No response

Expected behavior

1. current_sprint should be null if sprint is null
2. sprint should be null if ''

dbt Project configurations

  # jira
  jira_database: fivetran_db
  jira_schema: jira
  jira_using_sprints: true
  jira_using_components: false
  jira_using_versions: true
  jira_include_comments: true

Package versions

  - package: fivetran/jira
    version: 0.12.2

What database are you using dbt with?

snowflake

dbt Version

> palm shell
Executing command `bash` in compose...
root@eeb4fa054be1:/app# dbt --version
Core:
  - installed: 1.4.5
  - latest:    1.4.5 - Up to date!

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Disabling models

We're erroring out on everything dependent on the sprint source table since we don't have a sprint table in our Fivetran-synced schema. Is there functionality built into this package to disable like Zendesk using_schedules: false?

[Bug] daily_history model and enhanced model producing different values for the same field name

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The main problem is that we are seeing different values for the same field (issue_type) between the jira__daily_issue_field_history table and the jira__issue_enhanced table. It appears that jira__issue_enhanced reflects the raw.jira.issue_type.name value, while daily_issue_field_history reflects the raw.jira.field_option.name value - possibly because both have the same ID.

The results of this query provide an example:

select 
    i.id as issue_id
    , t.id::string AS issue_type_id
    , t.name AS issue_type_name
    , fh.value AS field_history_value
    , fo.id as field_option_id
    , fo.name AS field_option_name
    , e.issue_type_id::string AS enhanced_issue_type_id
    , e.issue_type AS enhanced_issue_type
    , d.issue_type AS daily_history_issue_type 
from raw.jira.issue i
left join raw.jira.issue_type as t
    on t.id = i.issue_type
left join raw.jira.issue_field_history fh
    on fh.issue_id = i.id
left join biz.dbt.jira__issue_enhanced e 
    on e.issue_id = i.id
left join biz.dbt.jira__daily_issue_field_history d 
    on d.issue_id = e.issue_id
left join raw.jira.field_option fo 
    on fo.id = i.issue_type
where i.id = 183458
    and fh.field_id = 'issuetype'
GROUP BY 1,2,3,4,5,6,7,8,9

Results:

ISSUE_ID	ISSUE_TYPE_ID	ISSUE_TYPE_NAME	FIELD_HISTORY_VALUE	FIELD_OPTION_ID	FIELD_OPTION_NAME	ENHANCED_ISSUE_TYPE_ID	ENHANCED_ISSUE_TYPE	DAILY_FIELD_HISTORY_ISSUE_TYPE
183458	10736	Delivery Site	10736	10736	Internal Quality Issue	10736	Delivery Site	Internal Quality Issue

According to @fivetran-avinash , this could be because Currently our data logic coalesces the field option name over the initial column name if it exists. [You can see this logic in action in Line 119 here.](https://github.com/fivetran/dbt_jira/blob/main/models/jira__daily_issue_field_history.sql#L119).

Relevant error log or model output

No response

Expected behavior

Ideally, we would be able to track both fields separately within the daily table. In other words, our end-users need to be able to filter on both Delivery Site issues and Internal Quality Issue issues separately. Also, ideally the issue_enhanced table values would match the daily_history table values for fields of the same name - right now the difference is causing some confusion.

dbt Project configurations

models:
jira:
schema: dbt
jira_source:
schema: dbt_stg

...

vars:
jira_database: raw
jira_schema: jira

jira:
issue_field_history_columns: [
'Activity'
,'Activity Date'
,'Affects versions'
,'Approved By'
,'Assignee'
,'Battery Serial Number'
,'BIS Link'
,'BIS Issue Type'
,'BIS Application'
,'BIS Root Cause'
,'Body Serial Number'
,'Cold Chain equipment type and model'
,'Complaint Source'
,'Complaint Type'
,'Components'
,'Containment Status'
,'Contributing factors during delivery'
,'Created'
,'Creation Reason'
,'Creator'
,'Critical Incident Count'
,'Current Customer Trust Level'
,'Customer Account'
,'Customer Issue Type'
,'Customer Type'
,'Customers Impacted'
,'Department'
,'Date and Time'
,'Delivery Issue Type'
,'Description'
,'Donors'
,'ECO #'
,'Epic Link'
,'Equipment Failure'
,'Equipment Status'
,'Exception Name'
,'Failure Category'
,'Failure Type'
,'Fix versions'
,'Flight ID'
,'Grounded Status'
,'Human injury caused by package'
,'Impact'
,'Impact on customer/patient'
,'Incident Timestamp'
,'Internal/External'
,'Issue Category'
,'Issue Category Label'
,'Issue Summary'
,'Issue Type'
,'Labels'
,'Location Relative to Nest'
,'Logs'
,'Loss product description'
,'LRU'
,'Make it Right Level'
,'Mission Stage'
,'Nest'
,'New Part Number'
,'Operator Report'
,'Order ID'
,'Order Types'
,'ORN'
,'OSD Checkboxes'
,'Other failure selection'
,'Package Drop location'
,'Package ID'
,'Parent Link'
,'Part Number'
,'People Reached'
,'Priority'
,'Product Risk'
,'Project'
,'Property damage caused by package'
,'Quantity'
,'Rank'
,'Released In'
,'Request participants'
,'Request Type'
,'Required Approvers'
,'Resolution'
,'Resolution Reason'
,'Resolved'
,'Root Cause'
,'Root Cause Category'
,'Root Failure'
,'Safety Performance Target Affected'
,'Satisfaction'
,'Serial Number'
,'Severity'
,'SLA'
,'Stakeholders'
,'Status Category Changed'
,'Story Points'
,'Sub-tasks'
,'Summary'
,'Supplier'
,'Time to approve normal change'
,'Time to first response'
,'Time to close after resolution'
,'Time to resolution'
,'Troubleshooting Results'
,'Unit ID'
,'Updated'
,'Viable Units'
,'Vehicle Generation'
,'Wing Serial Number'
,'Yes/No'
,'Yes/No/Unsure'
,'Zip Number'
,'Zone'
]

Package versions

packages:

• package: fivetran/jira
  version: 0.14.0

What database are you using dbt with?

snowflake

dbt Version

Core:

• installed: 1.3.1

Additional Context

Initial convo about the issue here: https://getdbt.slack.com/archives/C01D1R2JLLA/p1693234610157279

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Bug] Status field being populated with field options

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

After upgrading from version >=0.8.0, and running a full refresh of the Jira data, the status field on both jira__daily_issue_field_history and jira__issue_enhanced is being populated with values from the jira.field_options table, instead of the status id it was previously, or a value from the jira.status table.

Relevant error log or model output

No response

Expected behavior

I would expect the status field to either contain the status_id, or the name from the jira.status table.

dbt Project configurations

version: '1.0.0'
config-version: 2
vars:
  jira:
    issue_field_history_columns: ['Work Type', 'Story Points', 'Squad']

Package versions

packages:

• package: fivetran/jira
  version: [">=0.8.0", "<0.9.0"]

What database are you using dbt with?

redshift

dbt Version

1.0.7

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Lookback window for Incremental Models

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Currently the jira__issue_field_history model (and it's upstream models) have significant incremental logic in order to handle the high volume of data when it comes to issue field history. However, the logic is built to assume all issue field updates are synced in order and that has been seen to not be the case. There are instances where issue field history records are synced "late" and may not come in for a day or so. This then results in an incorrect end model as the incremental strategy assumes all records are synced in order.

Therefore, we should apply a lookback window within the incremental logic to ensure records synced late are captured in subsequent incremental runs. This will result in an impact to the performance of the models. However, if we do this right, it will be insignificant and more accurate which is the route we would prefer.

Describe alternatives you've considered

Running a full refresh periodically to capture these late arrival records.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

QUESTION - [issue when upgrading packages]

Are you a Fivetran customer?
Yes ! Marie Sergue, Data Analyst at Aircall

Your Question
I am trying to upgrade jira_source and jira packages but my CI test returns the following error:
Database Error in model int_jira__issue_calendar_spine (models/intermediate/field_history/int_jira__issue_calendar_spine.sql) Could not find parent table for alias "staging.ci_schema_stg_jira.stg_jira__issue".

Additional context

Please indicate the level of urgency and business impact of this request
Not super urgent but it's blocking me from fixing issues that were supposed to be fixed by the new package version.

[Feature] Add databricks compatibility

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Add databricks compatibility

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

BUG - ): SQL compilation error: ambiguous column name '_FIVETRAN_SYNCED' compiled SQL at target/run/jira/models/jira__issue_enhanced.sql

Are you a current Fivetran customer?

My name is Nataliya Babich, I am BI Developer at Nice,
Nice is Fivetran customer.

Describe the bug

I updated our version JIRA fivertran dbt package to version 0.3.0.
After run dbt run --full-refresh I get error in model jira__issue_enhanced.

ERROR creating table model jira.jira__issue_enhanced........ [ERROR in 1.65s]
2021-08-01 14:05:35.133777Z: Finished running node model.jira.jira__issue_enhanced
Database Error in model jira__issue_enhanced (models/jira__issue_enhanced.sql)
002028 (42601): SQL compilation error:
ambiguous column name '_FIVETRAN_SYNCED'
compiled SQL at target/run/jira/models/jira__issue_enhanced.sql

Steps to reproduce
I updated our version JIRA fivertran dbt package to version 0.3.0 (https://hub.getdbt.com/fivetran/jira/latest/),
then I ran dbt deps.
I added var jira_issue_history_buffer: 12 to dbt_project.yml.
I run dbt run --full-refresh.
In jira__issue_enhanced I get error

ERROR creating table model jira.jira__issue_enhanced........ [ERROR in 1.65s]
2021-08-01 14:05:35.133777Z: Finished running node model.jira.jira__issue_enhanced
Database Error in model jira__issue_enhanced (models/jira__issue_enhanced.sql)
002028 (42601): SQL compilation error:
ambiguous column name '_FIVETRAN_SYNCED'
compiled SQL at target/run/jira/models/jira__issue_enhanced.sql

Expected behavior

Project variables configuration

name: 'my_new_project'
version: '1.0.0'
config-version: 2
vars:
jira_issue_history_buffer: 12
issue_field_history_columns: ['Story Points','priority','summary','Team','Closed','Capitalize','Regression','Epic Story Points','Requirement Category','_FIVETRAN_SYNCED', 'Found By']
jira_database: FIVETRAN_DATABASE
jira_schema: JIRA
tfs_schema: DWH_JIRA
field_option_local: "{{ source('bi_jira', 'field_option') }}"
version: "{{ source('bi_jira', 'version') }}"
tfsdwhdatatotableau: "{{ source('bi_tfs', 'tfsdwhdatatotableau') }}"
tfs_history_data: "{{ source('bi_tfs', 'tfs_history_data') }}"

Package Version

packages:

• package: fivetran/jira
  version: 0.3.0

Warehouse

• BigQuery
• Redshift
  [x] Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context

Screenshots

Please indicate the level of urgency

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Bug] `jira__daily_issue_field_history` incorrectly joins component fields to `jira.field_option` instead of `jira.components`

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

When you include Components (or other multi-select fields that do not store options in jira.field_option but rather in their own special tables like jira.component), jira__daily_issue_field_history incorrectly tries to expand the field values by joining on field_option for all fields.

Relevant error log or model output

No response

Expected behavior

The models should handle the "special case" fields like component and join on the correct table.

dbt Project configurations

vars:
  jira:
    issue_field_history_columns:
      # - ...
      - "Components"
      # - ...

Package versions

jira 0.8.0

What database are you using dbt with?

bigquery

dbt Version

1.0.0

Additional Context

https://getdbt.slack.com/archives/C01D1R2JLLA/p1670436806950859

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

document `jira_issue_history_buffer` variable

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

we have this variable for extending the history of an issue past its closing date and we don't document it anywhere!

https://github.com/fivetran/dbt_jira/blob/main/models/intermediate/field_history/int_jira__issue_calendar_spine.sql#L73

Relevant error log or model output

No response

Expected behavior

users should know it exists and how to use it

dbt Project configurations

na

Package versions

na

What database are you using dbt with?

postgres, redshift, snowflake, bigquery, databricks

dbt Version

na

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] split out fields with the same name into distinct columns in daily_issue_field_history

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Address something like #90 by splitting fields with identical names.

So, if you had multiple 'sprint' fields, they would be surfaced separately (instead of coalesced) as their own columns in the daily issue field history model. These columns could then be combined by the user outside of the package. Perhaps we can make each column by unique be concatenating the field name with the unique field id (so sprint_10020, sprint_10029)

This is more ideal than the jira_field_grain variable solution because what if the two distinct-but-identical fields have concurrently active field values? Which do you choose? Do you aggregate? etc etc Lots of considerations that are unnecessary if we just keep these actually-distinct fields distinct

Describe alternatives you've considered

keeping the jira_field_grain variable

aggregating potentially totally-different field values together

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

BUG - the file:int_jira__issue_join failed on ambiguous column name 'ISSUE_ID'

Hi

We use with Jira package: fivetran/jira, version: 0.5.0
As of September 17th the DBT job failed on the file: int_jira__issue_join
With an error message: ambiguous column name 'ISSUE_ID' .
I investigated this file, and I see that in the cte : issue, the field: issue_id appears twice.
In the cte: join_issue there are joins to this field, and I think that the failure is because that.

this is the code of cte: issue

with issue as (

-- including issue_id in here because snowflake for some reason ignores issue_id,
-- so we'll just always pull it out and explicitly select it


select
    **issue_id,**  --first
    coalesce(revised_parent_issue_id, parent_issue_id) as parent_issue_id,

    "ORIGINAL_ESTIMATE_SECONDS",

"REMAINING_ESTIMATE_SECONDS",
"TIME_SPENT_SECONDS",
"ASSIGNEE_USER_ID",
"CREATED_AT",
"CREATOR_USER_ID",
"ISSUE_DESCRIPTION",
"DUE_DATE",
"ENVIRONMENT",

"ISSUE_ID", --second

"ISSUE_TYPE_ID",
"ISSUE_KEY",
"PARENT_ISSUE_ID",
"PRIORITY_ID",
"PROJECT_ID",
"REPORTER_USER_ID",
"RESOLUTION_ID",
"RESOLVED_AT",
"STATUS_ID",
"STATUS_CHANGED_AT",
"ISSUE_NAME",
"UPDATED_AT",
"WORK_RATIO",
"_FIVETRAN_SYNCED",
"REVISED_PARENT_ISSUE_ID",
"ISSUE_TYPE",
"PARENT_ISSUE_TYPE",
"PARENT_ISSUE_NAME",
"PARENT_ISSUE_KEY",
"IS_PARENT_EPIC"

from BIDEVDB.dwh_jira.int_jira__issue_type_parents

)

Did you do something with this code? we run with this version some weeks, and all was okay, what happened suddenly?
Can you check it asap and update us?
Now our data don't updated, and our users need it...

Thank for your quick check and response.

Rivki

[Bug] Status field populated with null in incremental running

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Hi

Regarding the field : status in table : JIRA__DAILY_ISSUE_FIELD_HISTORY.
When I run full running the field populate fantastic, but when I run incremental running the field populate with null values.
I investigate the code and see that in the high version of package(version: 0.9.0) the value in the table is status_name, in the low version(version: 0.6.0) the value is status_id, and when the value is status_id the problem not happen, because in high version the join is between status_id and status_name:

In this table status_id:

select *
from BIPRODDB.dwh_actimize_jira.int_jira__field_history_scd
where valid_starting_on >= (select max(date_day) from BIPRODDB.dwh_actimize_jira.jira__daily_issue_field_history )

In this table status_naem:

select *
from BIPRODDB.dwh_actimize_jira.jira__daily_issue_field_history
where date_day = (select max(date_day) from BIPRODDB.dwh_actimize_jira.jira__daily_issue_field_history )

And if the value arrived from the second table(most_recent_data) and join to status table with status_id, it will be null...

Relevant error log or model output

No response

Expected behavior

I expect to get the last value of status and not null, need to change or remove the join to status table, in cases that the value arrived from target table and not from the increment table.

dbt Project configurations

name: 'actimize_project'
version: '1.0.0'
config-version: 2

vars:
  # jira_issue_history_buffer: 12
  issue_field_history_columns: ['acceptance owner','acceptance criteria','action completed date','actual delivery date','actual pi','aggregated done story points','aggregated remaining story points',
                                'aggregated story points','aha url','analysis status','append release notes','artifact id','artifact version','artifactory link','assignment type','automated',
                                'backlog rank','begin date','bug origin','candidate for automation','capitalize','clone','closed','code freeze date','commitment level','commitment type','cs task id',
                                'cs task owner','customer','customer facing guides - docenter','customer facing guides - sandbox','customer impact','customer priority','documentation completed',
                                'documentation link','documentation required','draft document url','end date','environment name','epic color','epic description','epic link','epic name','epic planned pi',
                                'epic status','epic story points','epic type','escape analysis category','eta given date','expected next update','external issue id','field readiness notes','fix notes',
                                'flag - comment added','flag - status changed','found by automation','found on build','fr status','functionality','functionality required version','gdpr files location',
                                'git link','guide names','guide page\section','guide version','high level estimate','impact','installation steps','instance sp','instance version','internal facing guides - docenter',
                                'internal facing guides - sandbox','investigation steps','issue color','justification','last status change','last updated by wiser','limitation reason','limited release',
                                'link to case/sr','link to cs task','link to task','milestone','old issue key','origin','original estimate','originated in version','os type','overall cst priority',
                                'packaging status','planned date','planned pi','planned release','possible root cause','priority customer','product','project type','quality reason','quality sub-reason',
                                'rank','regression','release highlights','release notes','release on scope','release on time','release type','reopen counter','requested date','requested pi','requested release',
                                'required actions','requirement category','resolution name','resolution type','resolved date','risk score','rn description','rn fix','rn summary','rollback instructions',
                                'root cause analysis','root cause for no automation','rtc id','run on build','sdc link','severity','shared attachment folder','short description','shortcut opportunity',
                                'shortcut opportunity description','solution required version','start date','steps to reproduce','story description','story points','support quality of escalation','svn commits',
                                't-shirt size','task classification','task id','team name','technical availability date','technical description \ limitation','tenant id','test coverage type','url to rtc',
                                'url to tracker','version','virus scan status','wa bug created','wiser attachments','wiser case / sr owner','wiser case/sr#','wiser parent creation date','wiser sysid','wiser task cancellation reason']
  
  jira_database: fivetran_actimize
  jira_schema: jira_cloud_prod

models:
  +copy_grants: true 
  jira:
      +schema: dwh_actimize_jira
  jira_source:    
      +schema: dwh_actimize_jira   

Package versions

packages:

• package: fivetran/jira
  version: 0.9.0

What database are you using dbt with?

snowflake

dbt Version

dbt Version : 1.3

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Bug] Incorrect Historical Value when Two Updates on the Same Day

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

int_jira__pivot_daily_field_history has an issue where there are two transactions on the same day one of which is null. The scenario is as follows.

• An issue has two transactions on the same day changing the sprint. Its initially null then is set to an assigned sprint.
• int_jira__daily_field_history changes Null values to the text 'is_null'
  case when field_value is null then 'is_null' else field_value end as field_value,
• int_jira__pivot_daily_field_history chooses the max value for any sprints. The text 'is_null' is greater than the actual sprint id and is used.

I imagine this affects all other historical fields where there are two transactions on the same day. The code should be changed to select the last transaction on a day and use those values over others. Its not as simple as ignoring the nulls as those could be the last transaction in certain cases. This is the offending code block.

pivot_out as (

    -- pivot out default columns (status and sprint) and others specified in the var(issue_field_history_columns)
    -- only days on which a field value was actively changed will have a non-null value. the nulls will need to 
    -- be backfilled in the final jira__daily_issue_field_history model
    select 
        valid_starting_on, 
        issue_id,
        max(case when lower(field_id) = 'status' then field_value end) as status,
        max(case when lower(field_name) = 'sprint' then field_value end) as sprint

        ,
            max(case when lower(field_name) = 'due date' then field_value end) as due_date
        ,
            max(case when lower(field_name) = 'labels' then field_value end) as labels
        ,
            max(case when lower(field_name) = 'reporter' then field_value end) as reporter
        ,
            max(case when lower(field_name) = 'resolution' then field_value end) as resolution
        ,
            max(case when lower(field_name) = 'resolved' then field_value end) as resolved
        ,
            max(case when lower(field_name) = 'story points' then field_value end) as story_points
        ,
            max(case when lower(field_name) = 'team' then field_value end) as team
        from daily_field_history

    group by 1,2
),

Relevant error log or model output

No response

Expected behavior

• The null value should be ignored if its an earlier transaction on that day. Only the last field_value should be selected.

dbt Project configurations

N/A

Package versions

N/A

What database are you using dbt with?

postgres

dbt Version

N/A

Additional Context

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Current_date snowflake function cause to lag in data

Hi

I have a question about lag of data in our Jira dashboards.
Until the hour 10 AM (Israel time), we can't see the data of current date, although that in Fivetran there is data of current date.
I investigated this issue and find that the int table :int_jira__issue_calendar_spine use with filter: "where date_day <= current_date".

The function current_date bring me the real date just starting from 10 AM (Israel time), and this is the reason of lag in our data.

First I will happy to understand what is the meaning of current_date, which time is it?(I'm sorry, I know that it is snowflake function, but I don't understand according by this function brings the date) .
And also I want to ask if there is an option to change it for UTC time (for example sysdate() function) or to give an option to users, to choose the desired time.

Thank a lot for your feedback, and have a nice week.
Rivki

Run full and get the error message: SQL compilation error

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Hi

I use with jira package version 0.13.0:

packages:

• package: fivetran/jira
  version: [">=0.13.0", "<0.14.0"]

When I run command "dbt run", the running finished successfully, but when I run command "dbt run --full-refresh" I get the error message:
"SQL compilation error: Object found is of type 'VIEW', not specified type 'TABLE'"
The error is from table: jira__daily_issue_field_history.
The table is created, but in the end of script, after table created, there is a script:
"drop table if exists BIDEVDB.dwh_actimize_jira.jira__daily_issue_field_history__dbt_tmp"
And from this script there is the full error message:

"
drop table if exists BIDEVDB.dwh_actimize_jira.jira__daily_issue_field_history__dbt_tmp cascade
16:15:28 Snowflake adapter: Snowflake query id: 01ad678f-0604-9d33-0011-9f07023e562e
16:15:28 Snowflake adapter: Snowflake error: 002203 (02000): SQL compilation error: Object found is of type 'VIEW', not specified type 'TABLE'.
16:15:28 Snowflake adapter: Error running SQL: macro drop_relation
16:15:28 Snowflake adapter: Rolling back transaction.
16:15:28 Timing info for model.jira.jira__daily_issue_field_history (execute): 16:10:40.733197 => 16:15:28.287234
16:15:28 On model.jira.jira__daily_issue_field_history: Close
16:15:28 Database Error in model jira__daily_issue_field_history (models/jira__daily_issue_field_history.sql)
002203 (02000): SQL compilation error: Object found is of type 'VIEW', not specified type 'TABLE'.
compiled code at target/run/jira/models/jira__daily_issue_field_history.sql
"
Can you check why?
Than a lot

Relevant error log or model output

drop table if exists BIDEVDB.dwh_actimize_jira.jira__daily_issue_field_history__dbt_tmp cascade
Snowflake adapter: Snowflake query id: 01ad678f-0604-9d33-0011-9f07023e562e
Snowflake adapter: Snowflake error: 002203 (02000): SQL compilation error: Object found is of type 'VIEW', not specified type 'TABLE'.
Snowflake adapter: Error running SQL: macro drop_relation
Snowflake adapter: Rolling back transaction.
Timing info for model.jira.jira__daily_issue_field_history (execute): 16:10:40.733197 => 16:15:28.287234
On model.jira.jira__daily_issue_field_history: Close
Database Error in model jira__daily_issue_field_history (models/jira__daily_issue_field_history.sql)
  002203 (02000): SQL compilation error: Object found is of type 'VIEW', not specified type 'TABLE'.
  compiled Code at target/run/jira/models/jira__daily_issue_field_history.sql

Expected behavior

In running of full refresh, this script unnecessary, because there is no a temporary table:

create or replace transient table BIDEVDB.dwh_actimize_jira.jira__daily_issue_field_history
copy grants as...

A temporary table exists only in incremental running.

dbt Project configurations

Name your project! Project names should contain only lowercase characters

and underscores. A good package name should reflect your organization's

name or the intended use of these models

name: 'actimize_project'
version: '1.0.0'
config-version: 2

vars:

jira_issue_history_buffer: 12

issue_field_history_columns: ['acceptance owner','acceptance criteria','action completed date','actual delivery date','actual pi','aggregated done story points','aggregated remaining story points',
'aggregated story points','aha url','analysis status','append release notes','artifact id','artifact version','artifactory link','assignment type','automated',
'backlog rank','begin date','bug origin','candidate for automation','capitalize','clone','closed','code freeze date','commitment level','commitment type','cs task id',
'cs task owner','customer','customer facing guides - docenter','customer facing guides - sandbox','customer impact','customer priority','documentation completed',
'documentation link','documentation required','draft document url','end date','environment name','epic color','epic description','epic link','epic name','epic planned pi',
'epic status','epic story points','epic type','escape analysis category','eta given date','expected next update','external issue id','field readiness notes','fix notes',
'flag - comment added','flag - status changed','found by automation','found on build','fr status','functionality','functionality required version','gdpr files location',
'git link','guide names','guide page\section','guide version','high level estimate','impact','installation steps','instance sp','instance version','internal facing guides - docenter',
'internal facing guides - sandbox','investigation steps','issue color','justification','last status change','last updated by wiser','limitation reason','limited release',
'link to case/sr','link to cs task','link to task','milestone','old issue key','origin','original estimate','originated in version','os type','overall cst priority',
'packaging status','planned date','planned pi','planned release','possible root cause','priority customer','product','project type','quality reason','quality sub-reason',
'rank','regression','release highlights','release notes','release on scope','release on time','release type','reopen counter','requested date','requested pi','requested release',
'required actions','requirement category','resolution name','resolution type','resolved date','risk score','rn description','rn fix','rn summary','rollback instructions',
'root cause analysis','root cause for no automation','rtc id','run on build','sdc link','severity','shared attachment folder','short description','shortcut opportunity',
'shortcut opportunity description','solution required version','start date','steps to reproduce','story description','story points','support quality of escalation','svn commits',
't-shirt size','task classification','task id','team name','technical availability date','technical description \ limitation','tenant id','test coverage type','url to rtc',
'url to tracker','version','virus scan status','wa bug created','wiser attachments','wiser case / sr owner','wiser case/sr#','wiser parent creation date','wiser sysid','wiser task cancellation reason']

jira_database: fivetran_actimize
jira_schema: jira_cloud_prod

models:
+copy_grants: true
jira:
+schema: dwh_actimize_jira
jira_source:
+schema: dwh_actimize_jira

Package versions

packages:

• package: fivetran/jira
  version: [">=0.13.0", "<0.14.0"]

What database are you using dbt with?

snowflake

dbt Version

dbt Version: 1.5 (latest)

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Bug] `Valid ending at` field is not updating all the time

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Hey

I have noticed that the valid ending at field does not always update. I am trying to use this field to work out the time in status for each status as it transitions through the workflow but there are lots of nulls in the dataset. All the tables where this field is used appear to be impacted. Interesting I have no nulls for the valid starting at field

Example issue where it has not populated properly.

Relevant error log or model output

No response

Expected behavior

I would expect as the issue transitions that the timestamps are updated but as you can see this is not happening consistently

dbt Project configurations

will need to request from data team

Package versions

will need to request from data team

What database are you using dbt with?

snowflake

dbt Version

will need to request from data team

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Enhancement: Refactor int_jira__issue_calendar_spine

int_jira__issue_calendar_spine right now relies on run_query, which fails when you run dbt compile before dbt run.

Look into how to address this issue, and push findings to our ZD and Hubspot packages as well as they use calendar spines/run queries.

test

DBT Package performance issue

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Hi ,
we have an issue with the amount of data sync that lately has increased dramatically. and cause snowflake error 300005 even with a very large compute resources.
we discussed it with dbt Support and they would like to have a call with fivetran support engineer about the Fivetran-dbt JIRA package.
they wonder if it is possible that the Fivetran package is not filtering out the data using IS_ACTIVE=FALSE?

see also our previous Fivetran support request #157901.
can we set a meeting and investigate this issue with you?
dbt support engineer who is responsible for this issue is: Rijesh - [email protected]

Thanks in advanced,
Mali Rodan and Tomer Nahum
[email protected]
[email protected]

Relevant error log or model output

11:48:18  Database Error in model jira__daily_issue_field_history (models/jira__daily_issue_field_history.sql)
11:48:18    000603 (XX000): SQL execution internal error:
11:48:18    Processing aborted due to error 300005:4035471279; incident 7371288.
11:48:18    compiled Code at target/run/jira/models/jira__daily_issue_field_history.sql

Expected behavior

complete run successfully

dbt Project configurations

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'rnd_project'
version: '1.0.0'
config-version: 2
vars:
  jira_issue_history_buffer: 12
  issue_field_history_columns: ['customfield_10122','assignee','customfield_10135','customfield_10227','customfield_10079','customfield_10099',
                                'customfield_10136','customfield_10094','customfield_10133','customfield_10138','customfield_10126',
                                'customfield_10137','customfield_10106','customfield_10125','customfield_10096','customfield_10127',
                                'customfield_10139','customfield_10288','customfield_10128','customfield_10130','priority','customfield_10294',
                                'customfield_10295','customfield_10298','customfield_10300','customfield_10055','customfield_10124',
                                'customfield_10131','customfield_10132','customfield_10129','customfield_10134','customfield_10121','reporter',
                                'customfield_10045','customfield_10057','customfield_10329','customfield_10038','summary','customfield_10098',
                                'customfield_10291','customfield_10293','customfield_10286','customfield_10090','customfield_10304',
                                'customfield_10068','customfield_10081','customfield_10046','resolution','customfield_10093',
                                'customfield_10215','customfield_10229','customfield_10237','customfield_10220','customfield_11577','customfield_11578',
                                'customfield_10243','customfield_10037','customfield_10051','timespent','duedate','customfield_10250','customfield_10104',
                                'customfield_10113','customfield_10077','customfield_10253','customfield_11583','customfield_11510',
                                'customfield_10041','customfield_11598','customfield_11576', 'customfield_11605',  'customfield_11602']
  
  jira_database: fivetran_cx
  jira_schema: jira_cx
  github_database: fivetran_cx
  github_schema: github_cx 
  tfs_schema: dwh_jira_cloud_cx
  field_option_local: "{{ source('bi_jira', 'field_option') }}"
  tfsdwhdatatotableau: "{{ source('bi_tfs', 'tfsdwhdatatotableau') }}"
  tfs_history_data: "{{ source('bi_tfs', 'tfs_history_data') }}"
  blackduck_database: blackduck
  blackduck_schema: dwh
  cvs_final: "{{ source('bi_blackduck', 'cvs_final') }}"

models:
  +copy_grants: true
  jira:
    +schema: dwh_jira_cloud_cx
    jira__daily_issue_field_history:
      +pre-hook: 
        - "{{ change_warehouse() }}"
  jira_source:    
    +schema: dwh_jira_cloud_cx  
  github:
    +schema: dwh_github
  github_source:    
    +schema: dwh_github     
  
  rnd_project:
    nice_jira:
      tmp:
        materialized: view
      materialized: table
      schema: dwh_jira_cloud_cx
      jira__daily_issue_field_current_new:
        +pre-hook: 
        - "{{ change_warehouse() }}"
      
    blackduck:
      tmp:
        materialized: view
      materialized: table
      schema: dwh_blackduck

    

# This setting configures which "profile" dbt uses for this project.
profile: 'default'

# These configurations specify where dbt should look for different types of files.
# The `source-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
seed-paths: ["data"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
    - "target"
    - "dbt_modules"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/ directory
# as tables. These settings can be overridden in the individual model files
# using the `{{ config(...) }}` macro.

Package versions

packages:

• package: fivetran/jira
  version: 0.7.0

• package: fivetran/github
  version: [">=0.5.0", "<0.6.0"]

What database are you using dbt with?

snowflake

dbt Version

dbt Version: 1.5

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Problem with the last version of DBT package: fivetran/jira

Hi

Attached file with explanation on the bug.

we upgrade the package: : fivetran/jira To the last version: 0.4.0. and have some issues:

1. I set in variable: jira_issue_history_buffer = 12, and I expect to get the data of closed issues - 12 months from resolved date.
   For example, I have an issue that closed at 02.02.2020, I want the data until 02.2021, but I get the data until today.
   Can you help me and guide me if need to declare an additional variable, or need an another one?

2. We use with variable : issue_field_history_columns , I set in this var: issue_field_history_columns: ['Story Points','priority','summary','Team','Closed','Capitalize','Regression','Epic Story Points','Requirement Category','_FIVETRAN_SYNCED', 'Found By']
   Most of the fields without values after I run the new package.
   I investigated the code and find that there is a new logic in file: int_jira__daily_field_history.sql in cte: limit_to_relevant_fields

The old code:

limit_to_relevant_fields as (

-- let's remove unncessary rows moving forward and grab field names
select
combined_field_histories.*

from combined_field_histories

where 
lower(field_name) in ('sprint', 'status' 
                            {%- for col in var('issue_field_history_columns', []) -%}
                            , {{ "'" ~ (col|lower) ~ "'" }}
                            {%- endfor -%} )

),

The new code:

limit_to_relevant_fields as (

-- let's remove unncessary rows moving forward and grab field names
select
combined_field_histories.*

from combined_field_histories

where lower(field_name) = 'sprint' -- As sprint is a custom field, we filter by field name only for sprint. All others are on field_id.
    or lower(field_id) in ('status' 
        {%- for col in var('issue_field_history_columns', []) -%}
            , {{ "'" ~ (col|lower) ~ "'" }}
        {%- endfor -%} )

),

I see the comment that all fields aren't custom(exept sprint), but most of the fields that I use are custom:

In the meantime I return to the old version , but we need urgently a new feature in the new version.

Can you check and update us asap?

Thank a lot for your support.
Rivki

Casting To INT64 Line causing test failure

We have a data product that brings in an external package - DBT FiveTran Jira but the line in the model below fails in our run instance because it is trying to cast a string value into an integer. I began to explore and it turns out our piece of data that offends the line is a URL link that can have a string value.

dbt_jira/models/intermediate/int_jira__issue_epic.sql

Line 15 in 688a1ab

cast(field_value as {{ dbt.type_int() }} ) as epic_issue_id,

Is there a way to disable this test? or modify the line to accept string values?

Thanks, Kishen

Create a model for issues' version history

As a data modeler, I would like a model of version history where each row describes when a version field was added to or updated for an issue, so that I could match match issues to version history.

At a minimum, a lookup table of issue id, version id, and updated_at fields would be super helpful. At a maximum, these three fields plus the remaining fields on the version table could be added.

---

Are you a Fivetran customer?
We are a very happy Fivetran customer 😁

Is your feature request related to a problem? Please describe.
This feature request would help create version history of issues.

Please indicate the level of urgency and business impact of this request
This feature request would be a nice-to-have for us.

Are you interested in contributing to this package?

• No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

Issue fields do not update if field updated one-month after resolved date

What happened

At some point a jira user updates a custom field on an issue in the jira app after the issue has already been resolved for more than one month. For example, the user realized after the issues was resolved that a custom field was mislabeled. That updated custom field will not be updated in int_jira__field_history_scd model or its dependencies even after a full refresh. I believe the same is true for non-custom fields that get updated one month after resolution date is set.

This issue causes models in the date warehouse to no longer reflect the true state of issue field history available inside Jira app itself. And downstream BI consumers are perplexed why Jira says one thing and data warehouse says another.

Steps to Reproduce

1. Find an issue resolved jira from more than a month ago
2. Update custom (or any) field after issue has been resolved
3. run dbt (even with --full refresh option)
4. Field will not be updated

What I expected to happen

I expected the issue field to be updated even if the issue has been resolved.

Notes from Investigation

I chased the issue down to the source model that was not updating. It actually is not the model int_jira__field_history_scd but this model's dependency: int_jira__issue_calendar_spine. https://github.com/fivetran/dbt_jira/blob/master/models/intermediate/field_history/int_jira__issue_calendar_spine.sql

The model int_jira__issue_calendar_spine relates every issue to every day it was open plus a buffer of one month, in case the issue is re-opened, as this model is materialized incrementally. So for each issue, this model creates a window of time in which the issue was open. That window of time spans from issue created date to one month after resolution date.

Any field updates made during this time, will be reflected in downstream models. However, any field updates made after this window will not be updated in downstream models.

Possible Solutions

One possible solution (suggested) could be changing Line 45 on int_jira__issue_calendar_spine.sql to use update_at field from stg_jira__issues model instead of resolved_at field. This change would widen the window to one month beyond when the issue was last updated in jira.

Another possible solution could be changing Line 60 on int_jira__issue_calendar_spine.sql to a longer period of time after the resolution date or to a user-defined period of time.

BUG - new field values not being reflected in `jira__issue_enhanced` on incremental runs

Are you a current Fivetran customer?

nope but this came from https://getdbt.slack.com/archives/C01D1R2JLLA/p1635355824026800

Describe the bug

on incremental runs of the package, if you have removed (or changed ? -- asking Jacob this...) a field value, the new value is not being reflected in the issue enhanced model

Steps to reproduce

1. create a ticket and add label
2. sync fivetran
3. execute incremental run
4. confirm the label value is there in jira__issue_enhanced
5. remove the label in jira
6. resync
7. execute incremental run
8. if you still see the old label value, run a full refresh and see if it works now

Expected behavior

the model should pick up the new changes on an incremental run, full refreshes or materializing as tables should not be required

Project variables configuration

vars:
  jira_database: raw
  jira_schema: jira_fivetran
  jira:
    # Field options can be found with this query:
    # select name from RAW.JIRA_FIVETRAN.FIELD;
    issue_field_history_columns: ['Labels', 'Key', 'Epic Name', 'Severity', 'Project']

  'dbt_date:time_zone': 'America/New_York'
  dbt_artifacts:
    dbt_artifacts_database: analytics
    dbt_artifacts_schema: dbt_artifacts
    dbt_artifacts_table: artifacts

Package Version

copy packages.yml here

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context

this line may be an issue -- we're subtracting 1 day from the wrong side i think (so like we should do this to max(date_day) from {{ this }} instead of when the field was updated_at . i think this would cause the package to ignore the most recent day of data on incremental runs.

otherwise, the problem may live in the pivot model or the final daily issue field history model

Screenshots

Please indicate the level of urgency

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Bug] choose latest-ending sprint in issue_enhanced if there are multiple sprints

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

When we first developed the package, we were under the impression that a Jira issue could only be associated with one sprint at a time. However, this is not the case. When a sprint is completed with an incomplete issue, the issue is automatically rolled over to the next sprint (but still associated with the old sprint as well).

Because we made this assumption, if an issue is associated with 3 sprints, we choose the sprint whose information to highlight in the issue_enhanced model kinda at random (they all have the same updated_at and we just order by that in the row_number window function).

So, if an issue is rolled over, we may still point to the outdated sprint in the issue_enhanced model. Instead, we should choose the latest sprint, using the sprint_started_at and/or sprint_ended_at fields to determine which sprint to highlight for the issue.

Also, we may want to prepend sprint-data in this model with current_sprint_ or active_sprint_ or something because there's an additional sprint field passed on from the daily issue field history model. This sprint field is a comma-separated list of all the related sprint ids. Could be worth differentiating them better. (also could be worth using the sprint names instead of ids)

Relevant error log or model output

Outdated sprint info inn `jira__issue_enhanced`

Expected behavior

Sprint info should reflect the latest sprint

dbt Project configurations

na

Package versions

na

What database are you using dbt with?

postgres, redshift, snowflake, bigquery, databricks

dbt Version

na

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Add user roles to jira__user_enhanced.sql

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Information about users' roles and groups could be added to jira__user_enhanced.sql.

Tables PROJECT_ROLE, PROJECT_ROLE_ACTOR, and USER_GROUP are in the ERD, are installed by most users, but not yet incorporated into the models.

Describe alternatives you've considered

Not including this infromation.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

[Bug] Column names flipped in int_jira__user_metrics

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Lines 42 & 43 of int_jira__user_metrics to be revisited as the column alias names appear to be flipped. While the code still runs, this creates an issue down the line with the columns output by jira__user_enhanced.sql, specifically avg_age_currently_open_seconds and avg_close_time_seconds.

Relevant error log or model output

Before: Below is the output when the model is run as it stands. When looking at output, it is most easily observable in rows where the count is 0 in one column and not 0 in the other. For example in row 2, the count_closed... = 1, however the value for avg_close_time_seconds is null, however, we would expect this to be a non-zero value. Further, in this row, we would expect the average to equal the median since there is only 1 closed task. See also rows 3 and 6 for similar issues.

After: After switching the variable names, we get the expected outcome. The value is not null, and the average is the same as the median.

Expected behavior

Code is updated.

dbt Project configurations

n/a

Package versions

0.8.2

What database are you using dbt with?

bigquery

dbt Version

1.2.0

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Bug] Include a lookback window in incremental strategy

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

A recent customer issue showing discrepancies across their source data and transforms was a result from missing or late arriving records in a data load. Therefore we should at add a lookback window to our incremental models to help capture that data in a given window.

https://docs.getdbt.com/best-practices/materializations/4-incremental-models#late-arriving-facts

Relevant error log or model output

No response

Expected behavior

All data is captured! 🙏

dbt Project configurations

na

Package versions

na

What database are you using dbt with?

other (mention it in "Additional Context")

dbt Version

na

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Many fields in the table: jira__daily_issue_field_history

Hi

We use with the variable: issue_field_history_columns to hold the fields we need for the model.
For each field the history is stored in the history table: jira__daily_issue_field_history.
when we add fields and need to refresh the model full, it takes a few hours.
We have tens or hundreds of fields, it is very very heavy and every period we have to increase the warehose for this and the Snowflake's costs increase significantly.

It's quite unnecessary because in practice we don't need the history for the most of fields, from our point of view if the jira__issue_enhanced table contained the current state, the history table is unnecessary for the most of fields.

But in practice it seems that the jira__issue_enhanced table is based on the history table and brings the last record from it.

Is it possible to save resources and make the jira__issue_enhanced table or another table to bring us the latest information without having to save the history for all the fields we want to use?

Thanks for your answer and support.

Rivki

Issue Versions Addition to Issue Enhanced

A customer (@curious-bryan) requested the jira__issue_enhanced model to include current enriched fields from the jira version table. This feature request would most likely require the following updates (and others possibly):

• Addition of the version table within the dbt_jira_source package.
  • 89% of users have this table
• Modeling to map an issue to it's current version. Jira docs on versions show that versions are typically set at the project level. However, individual issue versions can be set and are logged within the issue_multiselect_history table as affectsVersions or fixVersions field_ids.

[Bug] Invalid column name in issue_field_history_columns

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

When using the issue_field_history_columns with a column having a leading numeric character in the name, the package fails. E.g. column name "1st reply date"

Relevant error log or model output

23:23:22  Database Error in model int_jira__pivot_daily_field_history (models/intermediate/field_history/int_jira__pivot_daily_field_history.sql)
23:23:22    001003 (42000): SQL compilation error:
23:23:22    syntax error line 29 at position 110 unexpected '1'.
23:23:22    compiled SQL at target/run/jira/models/intermediate/field_history/int_jira__pivot_daily_field_history.sql

Expected behavior

I expect the packages to succeed and create a column for the custom field in the jira__issues_enhanced and jira__daily_issue_field_history models with the corresponding values from the custom field.

dbt Project configurations

vars:
  # jira settings
  jira_database: <database>
  jira_schema: jira
  jira_using_sprints: false
  jira_using_components: false
  issue_field_history_columns: [
    '1st reply date'
  ]

Package versions

  - package: fivetran/jira
    version: 0.8.2
  - package: fivetran/jira_source
    version: 0.4.2

What database are you using dbt with?

snowflake

dbt Version

1.2.0

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

FEATURE - use dbt_utils.slugify() to use issue field names instead of IDs as column names

Are you a Fivetran customer?

fivetran made issue

Is your feature request related to a problem? Please describe.

not a problem per se, but a suboptimal solution to a previous problem regarding daily issue history fields. we decided to use field_ids to avoid special characters in column names in this PR #25

having columns called customfield_[id_number] isn't very user friendly, and i imagine users are renaming these columns as their actual field_names on top of the package models

Describe the solution you'd like

it would be great to have the columns named as their respective field.name instead of their ID. we can use dbt_utils.slugify() for this i think https://github.com/dbt-labs/dbt-utils/blob/main/macros/jinja_helpers/slugify.sql

Describe alternatives you've considered

• using IDs instead of potentially problematic field names

Additional context

Please indicate the level of urgency and business impact of this request

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
• No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

[Feature] Add issue_watcher to jira__issue_enhanced.sql

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Information about issue watchers (list of watchers or count, for example) could be added to jira__issue_enhanced.sql.

Table ISSUE_WATCHER is in the ERD, is installed by most users, but not yet incorporated into the models.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

[Bug] Sprint column not correctly mapped

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The Sprint field is not pulled through in int_jira__daily_field_history

If you refer to the following commit comment you will see the bug introduction
a1d5865#r76693371

Relevant error log or model output

No response

Expected behavior

Sprint history should be pulled through

dbt Project configurations

default is fine in this case

Package versions

0.8.1

What database are you using dbt with?

redshift

dbt Version

1.0.8

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

Adding default column names to issue_field_history_columns causes column ambiguity in jira__issue_enhanced

If you add a column that exists in in the jira issue source table to the issue_field_history_columns var, it results in column ambiguity in jira__issue_enhanced.

Example columns added to issue_field_history_columns that cause column ambiguity: "Due Date", "Environment", "Issue Type", "Work Ratio".

Currently, our workaround has been to exclude jira__issue_enhanced and downstream models in order to keep our dbt run command passing