Comments (13)
I've run through various tests (adding/removing/updating certificates/content-hosting-configurations) and the M3 interface appears to work. There's a bug with passing the certificate filename to the nginx configuration that needs fixing. The only operation I have left to look at is the cache purge operation, which at present will return a Not Implemented
error.
From what I remember from previously reading about nginx and purging cache entries is that this is done by handling PURGE requests and redirecting to a local nginx service which will expire the cache objects. As I remember it this operates either on specific cache entries or on the whole cache at once. I need to do some reading up on current nginx configuration to see how fine grained these cache purges can be made to be as I suspect we won't be able to provide the full level of control specified in TS 26.512.
from rt-5gms-application-server.
It would be worthwhile investigating whether Apache, Varnish, etc. offer the flexibility to purge entries based on a regular expression as well as Nginx, @davidjwbbc.
If this requirement of TS 26.512 cannot easily be met by existing caching web proxies, we should inform 3GPP SA4 and ask them to think again about this feature.
from rt-5gms-application-server.
From reading up on the uvicorn github issues about HTTP/2 support, it looks like simply switching from using the uvicorn module to the hypercorn module will enable support for HTTP/1.1 and HTTP/2 (and optionally HTTP/3 if the hypercorn[h3] module installation option is used). I'll modify my current solution to change to the hypercorn python module as the ASGI server.
from rt-5gms-application-server.
So using hypercorn, the M3 interface now supports HTTP/1.1, HTTP/2 upgrade and HTTP/2 prior knowledge.
from rt-5gms-application-server.
I've been looking at cache purging facilities available with different web server/proxy softwares. This is in order to understand what's possible with each one with respect to implementing the cache purging strategy defined in the 5G specifications.
Here is a table of 4 projects and their cache purging capabilities:
Software | Purge Object | Purge Whole Cache | Purge Cache Subtree | Purge Cache By Regex |
---|---|---|---|---|
NGINX | ⛔1 | ⛔1 | ⛔1 | ⛔1 |
NGINX Plus | 👍 | 👍 | 👍 | ⛔ |
Varnish | 👍 | 👍2 | ⛔ | ⛔ |
Apache HTTPd | 👍3 | 👍3 | 👍3 | 👍3 |
Squid | 👍 | 👍2 | ⛔ | ⛔ |
All pieces of software are able to purge individual items from their cache.
Two of the web proxies can purge the whole cache while running, two require the daemon to be stopped and the cache deleted.
NGINX can purge a subtree of cache items by using a wildcard symbol (*
) at the end of a cache key prefix, all items with the same prefix will be purged. To do this in apache the htcacheclean
command can be used to list all cache entries, filter those that start with a given prefix and use the htcacheclean
command to delete each one individually. There is a note on the Squid FAQs suggesting they would like to implement purge by wildcard but that this is not currently available.
Only Apache HTTPd can be made to purge by regular expression by using the htcacheclean
command to list all entries, filter the list using the regular expression and then use htcacheclean
to delete each item which matched the filter.
Footnotes
-
Not available as part of the software but can be emulated by using an external script to filter and delete cache files and then "reloading" the daemon. ↩ ↩2 ↩3 ↩4
-
Achieved by stopping the proxy daemon, deleting the cache and restarting the daemon. ↩ ↩2
-
Using the external
htcacheclean
command to list all cache entries and/or purge individual cached objects. ↩ ↩2 ↩3 ↩4
from rt-5gms-application-server.
Implemented the purge operation in the Python wrapper script as detailed above. The operation ends in a HUP
signal to NGINX if any cache files have been deleted.
Tested this using the test m3_client_cli.py
command in both purge all and purge regex modes of operation. Cache purged successfully.
I will let @devbbc do some testing against the rt-5gms-application-function with M3 interface. If no problems found then I'll tidy up the commits and generate a PR.
from rt-5gms-application-server.
Tested the working of the M3 interface by running this Application Server against the M3 client enabled rt-5gms-application-function. No problems found in this test and they worked as expected.
from rt-5gms-application-server.
Added M3 interface to the application server on my fork (feature-add-m3-interface branch), and am just running through some tests using curl to mock up client requests and view the responses.
Looking good so far.
from rt-5gms-application-server.
During testing of 5G-MAG/rt-5gms-application-function#7, it was noticed that the application server currently only handles HTTP/1.1 for its M3 service. Need to look into how the Python server can support HTTP/2 with prior knowledge on this interface.
from rt-5gms-application-server.
Thanks for the detailed research above, @davidjwbbc.
For the purposes of our reference implementation of the cache purge operation using Nginx, I suggest that we support a very limited subset of regular expressions. I have amended the issue description to specify a minimal subset corresponding to Nginx capabilities. (Implementations using different web proxies may implement a different subset of regular expressions, of course.)
from rt-5gms-application-server.
I've had an issue implementing the cache purge for NGINX.
So the part I missed when researching the table above is the fact that the cache purger in NGINX is only available in the commercial version (even though the documentation is included in the free version), the free version doesn't have a cache purging option built in.
A little looking around the internet suggests other people simply grep for the URL they wish to purge in all files in the cache directory and delete any matching files, then signal NGINX to reload its files. It's not a very nice way of doing it and relies on NGINX storing the cache key in plain text in the cache object files. But this is something we could do in Python. If we're going down this route then this also gives us the freedom to perform the operation using full regular expressions.
from rt-5gms-application-server.
To implement manually we can look for [LF]KEY: <cache-key>[LF]
near the start of the cache file (string appears 336 bytes into the file in the NGINX cache objects I've looked at). Reading the first 4KB of the file and looking for the cache key should do it. If we make sure the cache key contains the provisioning session id followed by the request URL path to the file being requested, then we can ensure that only entries for the correct session are removed. This can be done by using the proxy_cache_key
directive in the nginx configuration.
from rt-5gms-application-server.
While looking at TS 26.512 for 5G-MAG/rt-5gms-application-function#15, I noticed that the cache should also be purged when a Content Hosting Configuration is destroyed at reference point M1 on the 5GMS AF (or if the parent Provisioning Session is destroyed resulting in the Content Hosting Configuration subresource being destroyed).
I've therefore implemented this so that the NGINX cache is purged of all items in the path of the Content Hosting Configuration when it is destroyed by the 5GMS AF via reference point M3.
from rt-5gms-application-server.
Related Issues (20)
- Use OpenAPI to bindings script from 5G-MAG/rt-common-shared repository. HOT 3
- MVP#1 Application Server installation issues/observations HOT 21
- asyncio.create_task(..., name=...) doesn't work with Python 3.7 HOT 1
- Error, following the build instructions HOT 5
- Application Server crash on certificate upload HOT 3
- Content Hosting Configuration handling uplift in 5GMSd AS HOT 5
- Incorrect error response status code for M1_ContentHostingProvisioning_purgeContentHostingCache operation HOT 3
- Documentation updates after review on 23 Jan 2023 HOT 5
- Enhancement: Return number of cache entries purged in M3 response
- MVP#2 Application Server installation/running issues/observations HOT 11
- CI/CD - Automated deployment of 5GMS AS to Linode instance HOT 1
- Implement Server Health Check endpoint
- Uplift to TS 26.512 v17.4.0: Update M3 interface with new ContentHostingConfiguration structure HOT 1
- Feature: Media Streaming Data Reporting at reference point R4 HOT 1
- 5gms-application-server reports `Path parameters cannot have a default value` HOT 3
- Upstream redirects are passed to client HOT 9
- Redirection fails with recent OpenResty version HOT 1
- change in pydantic v2 HOT 3
- Development branch: Missing Nginx mime.types after clean installation HOT 2
- Updating PATH for launching Openresty HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rt-5gms-application-server.