airflow-plugins / mongo_plugin

mongo_plugin

mongo_plugin's Issues

Airflow UI doesn't have Mongo connection type

I'm using your plugin to automate data workflow. However, besides default connection type, it seems that airflow doesn't support mongo connection type. Is there any way to overcome this?

Add Operator for Azure Blob

Add docs!

PyMongo Query Doesn't Accept String Date Range

Hi,

When I was working with a query like this

      'mongo_query': [{ "$match": { "updated_at": {
                        "$lt": '{{ next_execution_date }}',
                        "$gte": '{{ execution_date }}'
                      } } },
                      { "$project": process_collection('orders')[0]}],

Those dates are parsed by Jinja and end up in string date, but PyMongo doesn't seem to recognize string dates parameters in aggregation. I manually transform to date object seems to work fine. I guess need to do some workaround here maybe? Since we're directly sending result template from Jinja here @@ ?

collection = mongo_conn.get_database(self.mongo_db).get_collection(self.mongo_collection)
results = collection.aggregate(self.mongo_query) if self.is_pipeline else collection.find(self.mongo_query)

Mongo to Google Storage / Big Query table?

Is it possible to have such operator?