Comments (8)
Fetching additional information automatically and using them to "enrich" the original recordset would break this assumption, since the new details would be unreviewed basically.
Understood and I guess that makes sense.
clients.yml
is generated during route server & switch configuration generation.
from arouteserver.
Hi @bluikko,
I think the tool already allows to deal with the use case you mentioned, but I might be wrong or maybe I misunderstood your request here.
When the clients-from-peeringdb
command is used to build the clients.yml starting from PeeringDB records, a file like this is generated:
asns:
AS112:
as_sets:
- AS112
AS15169:
as_sets:
- RADB::AS-GOOGLE
clients:
- asn: 112
ip:
- 192.0.2.1
- 2001:db8::1
- asn: 15169
ip:
- 192.0.2.2
- 2001:db8::2
When the command ixf-member-export
is executed and instructed to use the clients.yml file generated above, this is what it produces:
{
"version": "1.0",
"timestamp": "2022-11-19T14:11:59Z",
"ixp_list": [
{
"ixp_id": 0,
"ixf_id": 1,
"shortname": "test",
"vlan": [
{
"id": 0
}
]
}
],
"member_list": [
{
"asnum": 112,
"connection_list": [
{
"ixp_id": 0,
"vlan_list": [
{
"vlan_id": 0,
"ipv4": {
"address": "192.0.2.1",
"as_macro": "AS112"
}
},
{
"vlan_id": 0,
"ipv6": {
"address": "2001:db8::1",
"as_macro": "AS112"
}
}
]
}
]
},
{
"asnum": 15169,
"connection_list": [
{
"ixp_id": 0,
"vlan_list": [
{
"vlan_id": 0,
"ipv4": {
"address": "192.0.2.2",
"as_macro": "RADB::AS-GOOGLE"
}
},
{
"vlan_id": 0,
"ipv6": {
"address": "2001:db8::2",
"as_macro": "RADB::AS-GOOGLE"
}
}
]
}
]
}
]
}
The as_macro
is present on the export file, and set to the values retrieved from PeeringDB.
How far is this from what you were asking for? Am I missing anything? Maybe you're encountering some sort of bug?
from arouteserver.
Am I missing anything? Maybe you're encountering some sort of bug?
You are understanding exactly right. I am not seeing the as_macro
fields in the result IX-F JSON file. Example for AS112:
{
"asnum": 112,
"name": "AS112",
"connection_list": [
{
"ixp_id": 1,
"vlan_list": [
{
"vlan_id": 0,
"ipv4": {
"address": "192.0.2.1"
}
},
{
"vlan_id": 0,
"ipv6": {
"address": "2001:db8::1"
}
}
]
}
]
},
I have only anonymized the IPv4/6 addresses in the above extract from the generated IX-F JSON.
I just noticed that the run to generate IX-F JSON prints only a few lines as opposed to few hundred lines from the run to generate RS config files which have several lines about PeeringDB etc.
Maybe the behavior I am seeing is due to how I am running arouteserver? First I run arouteserver to generate each RS BIRD config. Then IX-F JSON is generated using docker image with something like:
arouteserver ixf-member-export --clients /root/clients.yml --merge-file /root/ix-f-staticdata.yml \
--ixp_id 1 -o /root/ix-f.json <IXPNAME> <IXPID>
I then added the arouteserver.yml
file after noticing that I passed it as a docker volume to the RS config generation but not to the IXP-F JSON generation which did not help. I could not see anything other obviously wrong or something to explain why ixf-member-export
would not query PeeringDB.
Since your example JSON includes as_macro
then I cannot at this point rule out the possibility that I am doing something wrong. I will try to look at my particular setup first before claiming it is a bug.
from arouteserver.
While working on troubleshooting this I noticed that currently run.sh
can only be used if none of the options need to be changed from default values - there is no way to pass arbitrary parameters to arouteserver
. For example I would like to pass --cache-dir
value to make sure I create a common docker volume for the several runs of arouteserver.
At least I would find it very useful to have some env var to pass arbitrary parameters, say CUSTOM_PARAMETERS
or whatever that's just added somewhere around line 195 of run.sh
.
from arouteserver.
Hi @bluikko, thanks for the additional information.
I now understand that you build your RS config files using the Docker container, and that you would also like to generate the Euro-IX file using the container as well: am I right?
Assuming this is the case, I see why you might want to have a way to customise the run.sh execution with additional options. However, I'd like to point out that while on one hand the Docker image can be used to easily get a "packaged" version of ARouteServer, with all the dependencies installed, on the other hand its entry point script (run.sh) is not really designed to offer a comprehensive way to interact with the tool. Nonetheless the docker image can still be used as a "wrapper" to easily run ARouteServer as if it was a local program, running on the host directly.
A simple Docker command like docker run -it --rm pierky/arouteserver:latest arouteserver
is enough to execute the program from within a container.
Starting from that, you can add additional options to docker run
and to arouteserver
to setup your environment, by mounting all the needed volumes to obtain what you need.
For example, this...
docker run \
-it \
--rm \
-v path-to-local/arouteserver.yml:/path-on-the-container/arouteserver.yml \
-v path-to-local/cache_dir:/path-on-the-container/cache_dir:w \
pierky/arouteserver:latest \
arouteserver COMMAND --cfg /path-on-the-container/arouteserver.yml
... would allow you to run ARouteServer using a custom arouteserver.yml file on the host, crafted in a way that custom paths on the container could be used for the cache directory for example. The file could be something like this...
arouteserver.yml
# ... other settings
cache_dir: /path-on-the-container/cache_dir
# ... other settings
... with cache_dir
pointed towards a directory that you mounted via docker run
.
Basically all the arouteserver xxx
commands that you see in run.sh can be executed in a similar way, directly from the host.
Please let me know what you think about the solution I'm proposing here.
I do see how a customization of run.sh would make things easier for your specific use case, but at the same time I'm a bit reluctant to implement it because I also see how difficult it could be to cover all the potential use cases that usually ARouteServer deal with by modifying the Docker image. It could easily become a "full mesh" of environment variables and CLI options, all condensed into a single Docker image and setup.
In addition to what I've proposed above, you can still edit the run.sh file and use your local version as the entry point for the container.
All this said, it's still unclear to me why your ixf-member-export
command is not adding the IRR info to the export file. If you could share the input clients.yml with me and also the exact command you run to have it translated into the Euro-IX JSON I could try to reproduce the issue on my side.
Thanks
from arouteserver.
Please let me know what you think about the solution I'm proposing here.
Yes I had thought about just passing the docker & arouteserver options I need to arouteserver bird
instead of relying on run.sh
. If you think some kind of "passing arbitrary switches" to run.sh
is not necessary there are other ways to accomplish the needful. I certainly see your point and I totally agree that a good vision is very important instead of trying to do everything suggested. Also I did not notice that cache_dir
can be passed in the configuration file so I'll use that then, it seems cleanest.
All this said, it's still unclear to me why your
ixf-member-export
command is not adding the IRR info to the export file. If you could share the input clients.yml with me and also the exact command you run to have it translated into the Euro-IX JSON I could try to reproduce the issue on my side.
I just found out a difficult-to-notice typo in the command, I will do some more tests tomorrow before bothering you more.
from arouteserver.
It took a while to get things tested again since I wanted to use the cache_dir
parameter. Due to various reasons that necessitated running arouteserver
(the Docker image) as non-root which also meant stopping using run.sh
.
The typo I mentioned in #113 (comment) was quite astonishing - I passed the PeeringDB API key to docker using -v
(volume) and not -e
(env var). I have no idea how it even worked in the first place. But fixing that did not make arouteserver ixf-member-export
to query PeeringDB.
It is run as (sanitized):
docker run \
-t --rm -u 42:42 \
-e SECRET_PEERINGDB_API_KEY=${Y} \
-v ${X}/custom-general.yml:/etc/arouteserver/general.yml:ro \
-v ${X}/arouteserver.yml:/etc/arouteserver/arouteserver.yml:ro \
-v ${X}/clients.yml:/etc/arouteserver/clients.yml:ro \
-v ${Z}/ix.json:/tmp/ix.json \
-v ${X}/static.yml:/tmp/static.yml:ro \
-v ${X}/cache:/tmp/arouteserver_cache \
${IMAGE} arouteserver ixf-member-export --cache-dir /tmp/arouteserver_cache \
--merge-file /tmp/static.yml --ixp-id 1 -o /tmp/ix.json IXPshort 42
And it displays no output. The generated ix.json
file is otherwise good but includes as_macro
for only those clients that have it defined in clients.yml
. Since there is no output whatsoever it seems obvious that PeeringDB is not queried at all. Other tasks (RS config, HTML policy, IRR AS-set) work correctly with the similar parameters for docker run
and other config.
I have been trying to see if I have some stupid problem somewhere but could not notice any so far.
As a side note: using cache_dir
dropped the typical total run time from 10 minutes to 5 minutes.
from arouteserver.
Hi @bluikko, thanks again for the details.
I think I'm starting to figure out what's going on: the ixf-member-export
is not designed to fetch information from PeeringDB at all. It just formats the same information that are inside the clients.yml file in a Euro-IX JSON compatible format.
Following from your very first comment in the opening of this issue, the workflow I thought you were using was something like clients-from-peeringdb
-> clients.yml -> ixf-member-export
-> Euro-IX JSON file. So, I was under the assumption that your clients.yml file was generated starting from PeeringDB, so that it contained all the expected as_sets
records, and that eventually those records were not included in the output Euro-IX JSON.
The fact that only the as_sets
records which are part of the clients.yml file are included in the JSON file is by design. Basically, the ixf-member-export
command does not infer any additional information before generating the JSON file. It does not "augment" the information already present on the clients.yml in any way. This is done on purpose, since the file that is generated in that way is expected to be an authoritative source of information for the IXP; so, clients.yml is assumed to be the source of truth, to contain information that are reviewed and approved by the IXP operator, and eventually only what's in there is dumped into the output JSON file. Fetching additional information automatically and using them to "enrich" the original recordset would break this assumption, since the new details would be unreviewed basically.
I hope this helps to clarify what you're seeing.
What I'd like to check with you is whether the workflow I've summarized above would fit your needs. You could use clients-from-peeringdb
to build your clients.yml, and then use that for both config generation and Euro-IX JSON export. You could run clients-from-peeringdb
to generate a "new" clients.yml file programmatically, like clients.yml.NEW, and then compare it with the official one (diff clients.yml clients.yml.NEW
or similar) and if you like the changes "promote" the new one to official (basically just a mv clients.yml.NEW clients.yml
, maybe after implementing backup and versioning if you desire them, via git
for example). In this way, with the addition of --merge-file
, you could be probably able to obtain the desired level of automation and customization. WDYT?
from arouteserver.
Related Issues (20)
- Allow configuration of the PeeringDB URL
- Add support for peeringdb-py
- __init__() got an unexpected keyword argument 'allowed_methods' after upgrade to 1.16 HOT 9
- Remove RGNET from whois sources HOT 1
- Route-Servers with Peer Active to ASNs that were excluded
- Double "AS" in comment for ASN without IPv6 in ARIN HOT 1
- Deprecate ARIN-WHOIS data source HOT 5
- feature request: add anchors to route server policy textual representation, other minor issues HOT 2
- Add routeserver attribute to IX-F export HOT 5
- Add metadata to policy textual representation
- Max prefix limit exceed shutdown with auto recovery HOT 1
- ipv4 table entry in bird config HOT 3
- PeeringDB AS-set name format should not be used in AS-set IRR object export HOT 6
- Looking Glass Client Configuration HOT 2
- Issue in IPv4 prefix length verification for RFC8950 HOT 3
- Double "AS" in comment for ASN without prefixes in Registro.br
- BUG - AS-SET filters starting with RS and other characters are not processed HOT 4
- Registro.BR dump parsing resiliency and error message clarity HOT 6
- PeeringDB API calls for `clients-from-peeringdb` done without API key HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arouteserver.