Comments (5)
First option - link to Atlas or other 3rd party
As @verdan points to Atlas already has strong lineage modelling and UI - a shortcut to navigating to that from Amundsen might simply be to populate the TABLE LINEAGE
sidebar link with URL data pointing to the right lineage page as shown here (this is what I understand Lyft is already doing with the 3rd party lineage tool they have deployed)
2nd option - "the younger brother" populate Neo4j
Another option which we could name "Atlas lineage's younger brother" would appear if you take a closer look at Amundsen's current Neo4j backend's data model. You'll notice the ingredients to model and store lineage are already in the diagram. It's the DAG
- write_to
-> Table
and Table
- replicated_from
-> Table
"links":
In fact you could claim that the - DAG
- writes_to
-> Table
part is already visualized by the GENERATED BY
section in the top screenshot in this comment. See further on near-term plans in https://github.com/lyft/amundsenfrontendlibrary/issues/202#issuecomment-504213401 [broken and outdated link]
Related issue in the frontend repo discussing "more ambitious" Amundsen lineage visualization: https://github.com/lyft/amundsenfrontendlibrary/issues/110
from amundsen.
@apoorva007 we haven't build the support in databuilder for neo4j as metadata backend. But I know @verdan 's team has built support for pushing lineage info into atlas which is another proxy offering for metadata.
cc @verdan for further comment.
from amundsen.
@apoorva007 you can use Amundsen with either Neo4j or Apache Atlas as the backend. We use Apache Atlas (and are constantly working on improving and adding new features in Amundsen) as our metadata proxy. Atlas supports the lineage out of the box, which is a one of a huge advantage of using Atlas as metadata proxy. It connects with many elements like spark, hive etc. using hooks and bridges, and thus records the whole lineage out of the box.
We have not yet started implementing lineage in Amundsen (via Atlas proxy), but please let us know what do you think of the this, your requirements, and if we can collaborate more on that part.
from amundsen.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
from amundsen.
2nd option above has moved a bit forward with merge of amundsen-io/amundsendatabuilder#126 as far as I can see this means with Neo4j #69 (comment) is essentially implemented so you can build a Databuilder job which will build lineage in Amundsen Metadata service.
Submitting a PR to ingest some sample data modeled after previous sample ingestion PRs seems like a good good first issue
if you have Python skills and want to get your hands dirty and move this feature forward sooner rather than later!
If you have React skills https://github.com/lyft/amundsenfrontendlibrary/issues/110 might be what you want to look at
Roadmap link
https://github.com/lyft/amundsen/blob/master/docs/roadmap.md#native-lineage-integration
from amundsen.
Related Issues (20)
- Elastic Search Loader fails for Root mapping definition has unsupported parameters error HOT 5
- Frontend npm install problems (lyft repository dependencies) HOT 11
- Broken images in `application_config` documentation HOT 6
- Lineage graph fails to render and due to "missing nodes" HOT 4
- GPL-licensed dependency `unidecode` HOT 5
- install Amundsen azure AKS HOT 3
- Amunsen crashes because yaml file has incompatible versions mentioned HOT 3
- Bug Report neo4j.exceptions.ServiceUnavailable: Couldn't connect to localhost:7687 trying to load sample data HOT 4
- Remove stale bot to allow address issues in a timely manner HOT 2
- Getting error when using sample_glue_loader.py HOT 1
- problem with setup.py HOT 3
- Dynamic notices not updating if we use nav bar search to open another table HOT 1
- Dynamic notices showing incorrect value(s) HOT 1
- neo4j_amundsen couldn't open temporary file HOT 3
- Helm Chart not deployable on k8 1.25 HOT 6
- Bug Report: BigQuery columns `sort_order` attribute is incorrect HOT 1
- Docker compose for docker-amundsen-atlas.yml is failing HOT 2
- Feature Proposal
- Add support for MaxCompute HOT 2
- Support for Unity Catalog HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amundsen.