Support for inline configuration strings about singer-python

[luandy64]

Couldn't you do:

echo '{"host": "mysql-host.com", \                                                                                                                                                                                   
       "port": "3306", \                                                                                                                                                                                             
       "user": "$USR_PROD", \                                                                                                                                                                                        
       "password": "$PWD_PROD"}'                                                                                                                                                                                     
     > /tmp/my_config; \                                                                                                                                                                                             
tap-mysql --config /tmp/my_config

[dcereijodo]

I totally can (at least in my use case). My suggestion was more about the convenience of it. When I have to do that inside an Airflow BashOperator for example, it starts looking odd

BashOperator(
  task_id='syn_animals',
  bash_command="""
    echo '{"host": "mysql-host.com", \                                                                                                                                                                                   
       "port": "3306", \                                                                                                                                                                                             
       "user": "$USR_PROD", \                                                                                                                                                                                        
       "password": "$PWD_PROD"}'                                                                                                                                                                                     
     > /tmp/my_config;                                                                                                                                          
   tap-mysql --config /tmp/my_config
  """
)

Also I might not have a good place (or a safe place) to put that temporal config file.

In this custom tap I experimented with an optional parameter --overrides which accepts a JSON string that gets merged into a default (or empty otherwise) configuration file. That would also work here.

BashOperator(
  task_id='syn_animals',
  bash_command="""
    tap-mysql --overrides '{"host": "mysql-host.com", "port": "3306", "user": "$USR_PROD", "password": "$PWD_PROD"}'
  """
)

But again it's just a matter of convenience :)

[luandy64]

Understood, feel free to open a PR for the feature 👍

On another note, how do you handle state files? Or do you not use those?

[dcereijodo]

Great. Will do that :)

No, I do not need state files. But if I were to implement that, what's wrong with using the same approach? So

BashOperator(
  task_id='syn_animals',
  bash_command="""
    tap-mysql \
      --config ~/.singer.io/tap_mysql_defaults.json \ # staging and prod configuration is identical, only the host changes
      --overrides '{"host": "$MYSQL_HOST"}' \
      --state '{"type": "STATE",  "value": {"bookmarks": {"db-animals": {"version": 1509135204169, "replication_key_value": "{{ ts }}", "replication_key": "updated_at"}}, "currently_syncing": null}}' \
    | target-redshift ...
  """
)

[luandy64]

Nothing wrong with it 😄 It occurred to me that either you are not using state at all or we would have to also implement this for state

[dcereijodo]

Right. I sent a PR that implements this behavior in the load_json function from the utils, so it should work for config, state and properties arguments.