GithubHelp home page GithubHelp logo

canonical / canonicalwebteam.discourse Goto Github PK

View Code? Open in Web Editor NEW
11.0 17.0 16.0 430 KB

License: GNU Lesser General Public License v3.0

Python 100.00%
package python-package python pypi pypi-package web-and-design

canonicalwebteam.discourse's Introduction

canonicalwebteam.discourse

Flask extension to integrate discourse content generated to docs to your website. This project was previously named discourse_docs.

Writing documentation

Documentation for how to write documentation pages in Discourse for consumption by this module and how to configure the website to use the module can be found in the Canonical discourse.

Example Flask template for documentation pages can be found in examples folder. Please refer to the README in that folder for more information.

Install

Install the project with pip: pip install canonicalwebteam.discourse

You can add the extension on your project as follows, replacing, at least, base_url and index_topic_id with your own settings:

import talisker.requests
from canonicalwebteam.discourse import DiscourseAPI, Tutorials, TutorialParser

app = Flask("myapp")
session = talisker.requests.get_session()

discourse = Tutorials(
    parser=TutorialParser(
        api=DiscourseAPI(
            base_url="https://forum.example.com/", session=session
        ),
        index_topic_id=321,
        url_prefix="/docs",
    ),
    document_template="docs/document.html",
    url_prefix="/docs",
)
discourse.init_app(app)

Once this is added you will need to add the file document.html to your template folder.

Local development

For local development, it's best to test this module with one of our website projects like ubuntu.com. For more information, follow this guide (internal only).

Running tests, linting and formatting

Tests can be run with Tox:

pip3 install tox  # Install tox
tox               # Run tests
tox -e lint       # Check the format of Python code
tox -e format     # Reformat the Python code

Instructions for Engage pages extension

Because you are viewing a protected topic, you must provide api_key and api_username. You also need an index topic id, which you can get from discourse.ubuntu.com. Your index topic must contain a metadata section. Visit the EngageParser for more information about the structure. You are encouraged to use an blueprint name that does not collide with existent blueprints. The templates must match the ones provided in the parameters indicated.

Here is an example of an implementation:

engage_pages = EngagePages(
    api=DiscourseAPI(
        base_url="https://discourse.ubuntu.com/",
        session=session,
        get_topics_query_id=14,
        api_key=DISCOURSE_API_KEY, # replace with your API key
        api_username=DISCOURSE_API_USERNAME, # replace with correspoding username
    ),
    category_id=51,
    page_type="engage-pages", # one of ["engage-pages", "takeovers"]
    exclude_topics=[] # this is a list of topic ids that we want to exclude from Markdown error checks
    additional_metadata_validation=[] # list of additional keys in the metadata table that you want to validate existence for e.g. language
)

In your project, you need to create your own views:

app.add_url_rule(
    "/engage", view_func=build_engage_index(engage_pages)
)

app.add_url_rule(
    "/engage/<path>", view_func=single_engage_page(engage_pages)
)
  • Where build_engage_index would be your view for the list of engage pages, which you can get by using the method EngagagePages(args).get_index()
  • While single_engage_page would be your single engage pages view, which you can get using EngagePages(args).get_engage_page(path)

Similarly for takeovers, you just need to pass page_type="takeovers".

  • To get a list of takeovers EngagePages(args).get_index() also.
  • To get a list of active takeovers EngagePages(args).parse_active_takeovers().

canonicalwebteam.discourse's People

Contributors

albertkol avatar anthonydillon avatar bartaz avatar carkod avatar codeempress1 avatar edlerd avatar jkfran avatar jpmartinspt avatar lukewh avatar nottrobin avatar petesfrench avatar renovate-bot avatar renovate[bot] avatar solazio avatar sowasred2012 avatar sparkiegeek avatar tbille avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

canonicalwebteam.discourse's Issues

Enable syntax highlighting

It would be nice if Markdown's syntax highlighting would work, e.g.:

```yaml
foo: bar
baz: ["foo"]
```
foo: bar
baz: ["foo"]

sitemap.xml doens't scale

Generating a sitemap is really expensive, every time the /sitemap.xml is loaded we make:

  • an API call to the index topic to get the list of URLs and topics.
  • and API call to the topic for EACH post to get the last modified date

This is really expensive when the amount of topics grows. For example, the engage pages have 278 discourse topics. This means we make 279 API calls (index topic + each topic) to generate the sitemap. The makes the engage page sitemap timeout constantly (https://ubuntu.com/engage/sitemap.xml).

To solve this we could paginate the sitemap:

  • the endpoint /sitemap.xml would become a sitemap index
  • we create the endpoint: /sitemap-<PAGE>.xml where PAGE is the page where 10(to define) topics are listed

This would make the sitemaps lighter, much faster to load and easier to parse in case we need to process them.

Update readme to include discourse index topic format

We have on the readme how to add the discourse module, but as we've seen with some community members using this module, most people don't know that we use a index topic that must contain a certain number of tables in a specific format for this discourse module to work.

Should we maybe add this to the readme?

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.


  • Check this box to trigger a request for Renovate to run again on this repository

Discourse updates preventative maintenance

As we saw with the upgrades on the kubernetes.io discourse, there are some changes to the way headings are generated in 2.7
There is some information on this here.

It would be really nice if we could leverage the automatic anchor headings feature in particular, so we didn't require everyone to manually create their own headings in html if they want to link to anything.

Need ability to create alternative main pages with navigation

For the purposes of testing alternative doc layouts, it would be extremely useful to be able to create alternative main pages (in a category not visible to users) which include navigation elements that differ from the public main doc page. Currently there is no obvious way to create another page that accepts the ## Navigation menu and presents it.

Getting a KeyError: 'rows' error with Discourse 2.8.0.beta7

Hi,

first, thank you for the great work and for sharing it!

I use Discourse 2.8.0.beta7 and run a flask app containing only the python module.

Running the app in debug mode, I get the following output:

[2021-11-08 13:49:12 +0100] [6548] [INFO] Starting gunicorn 20.0.4
[2021-11-08 13:49:12 +0100] [6548] [INFO] Listening at: http://127.0.0.1:8000 (6548)
[2021-11-08 13:49:12 +0100] [6548] [INFO] Using worker: sync
[2021-11-08 13:49:12 +0100] [6549] [INFO] Booting worker with pid: 6549
track API called when there is no active context, data will not be stored
[2021-11-08 13:49:15,204] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/alex/.local/lib/python3.9/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/app.py", line 238, in document_view
    document = self.parser.parse_topic(self.parser.index_topic)
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/parsers/tutorials.py", line 41, in parse_topic
    self.tutorials = self._get_tutorials_topics()
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/parsers/tutorials.py", line 112, in _get_tutorials_topics
    response = self.api.get_topics(topics)
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/models.py", line 64, in get_topics
    return response.json()["rows"]
KeyError: 'rows'

If I understand it properly, the troubles come from line 57 to 62, where the API call takes place:

        response = self.session.post(
            f"{self.base_url}/admin/plugins/explorer/"
            f"queries/{self.get_topics_query_id}/run",
            headers=headers,
            data={"params": f'{{"topics":"{topics}"}}'},
        )

The respective logs from my server (404 Code) indicate that the call is unsuccessful:

X.X.X.X - - [08/Nov/2021:14:10:38 +0100] "POST /admin/plugins/explorer/queries/None/run HTTP/1.0" 404 99 "-" "python-requests/2.26.0"

If I am not mistaken, I require the Data Explorer Plugin for discourse to make it work. I installed it and would guess I need to create a query for Data Explorer? Could somebody help me out with a template or something?

I would suspect, that the answer is somewhere in the TutorialParser (tutorial.py), but I cannot get my head around it.

Thank you in advance!

exported html link misses trailing "

Hi,
I found a bug processing https://discourse.ubuntu.com/t/service-ldap-replication/15508/1
As downloaded by https://github.com/canonical/webteam-jenkins-scripts/blob/main/server-guide-pdf/download-server-docs.py
which utilizes this code here.

Markdown in discourse:

the database suffix are the same, and [enable TLS](https://discourse.ubuntu.com/t/service-ldap-with-tls/11578#heading--certs-for-consumer).

Becomes this in the exporter

the database suffix are the same, and <a href="https://discourse.ubuntu.com/t/service-ldap-with-tls/11578#heading--certs-for-consumer>'enable TLS</a>

Please notice the missing " in the Link

<a href="https://discourse.ubuntu.com/t/service-ldap-with-tls/11578#heading--certs-for-consumer>'
should be
<a href="https://discourse.ubuntu.com/t/service-ldap-with-tls/11578#heading--certs-for-consumer">'

Empty lists in side nav when pages have no level

In multipass after adding some hidden pages with no level the side navigation started showing a line, which is a result of many empty lists added to the navigation:

image

Part of the config:

| 1 | reference | [Reference](/t/27144) |
| 2 | alias | [Alias](/t/28409) |
| 2 | driver | [Driver](/t/28410) |
| 2 | host | [Host](/t/28493) |
| 2 | instance | [Instance](/t/28469) |
| 2 | mount | [Mount](/t/28470) |
| 2 | multipass-cli-client | [Multipass CLI client >](/t/28472) |
| 3  | multipass-cli-commands| [Multipass CLI commands](/t/29329) |
| 2 | multipass-gui-client | [Multipass GUI client](/t/28484) |
| 2 | platform | [Platform](/t/28491) |
| 2 | service | [Service](/t/28494) |
| 2 | settings-explanation | [Setting >](/t/27347) |
| 3 | get-and-set-keys | [`get` and `set` keys](/t/29326) |
| 1 | explanation | [Explanation](/t/27145) |
| 2 | exec-shells | [`multipass exec` and shells](/t/27441) |
||||
|  | | `multipass` commands |
|  | alias-command | [alias](/t/24129) |
|  | aliases-command | [aliases](/t/24131) |
|  | authenticate-command | [authenticate](/t/26500) |
|  | delete-command | [delete](/t/27322) |
|  | exec-command | [exec](/t/10851) |
|  | find-command | [find](/t/8351) |
|  | get-command | [get](/t/13735) |
|  | get-command-preview | [get (preview)](/t/27345) |
|  | help-command | [help](/t/8349) |
|  | info-command | [info](/t/10848) |
|  | launch-command | [launch](/t/10846) |
|  | list-command | [list](/t/10847) |
|  | mount-command | [mount](/t/27212) |

https://discourse.ubuntu.com/t/multipass-documentation/8294

500 error when navigation table has rows without level

Not sure if it's a bug or "feature", but seems that having navigation table items without level between items that have level breaks the nav parsing.

[details=Navigation]
| Level | Path | Navlink |
| -- | -- | -- |
| 1 | test1 | [Heading1](https://discourse.ubuntu.com/t/how-to-write-and-format-discourse-specs/27036) |
| 1 | test2 | [Heading2](https://discourse.ubuntu.com/t/segmented-control-component/26835) |
| | test-hidden | Test |
| 2 | test3 | [Heading2](https://discourse.ubuntu.com/t/segmented-control-component/26835) |
| 3 | test4 | [Heading2](https://discourse.ubuntu.com/t/segmented-control-component/26835) |
| 3 | test5 | [Heading2](https://discourse.ubuntu.com/t/segmented-control-component/26835) |
[/details]

Having something like this (having items with level above 1 after empty level) breaks navigation parsing with 500 error.

Screenshot 2022-06-21 at 15 00 07

I don't know if rows without level have any special meaning, but maybe they should just be ignored, and not make the navigation parsing break?
Or is this expected to have level 0 or 1 after empty level for some reason?

Allow all pages in the URL mapping to render

I suspect it's because it's in the "Charming docs" category rather than the "Docs" category directly. Discourse docs does a check that the topic is in the expected category and only presents it on the site if it is, otherwise it redirects to the forum. This is to prevent people editing topics (which is open to most registered users) to add links to other topics that aren't in the curated docs category. However, it would almost certainly be worth extending the permission to sub-categories.

I'm not married to this mechanism though, since we now also have a URL mapping list in the index topic (this didn't exist when the above mechanism was first added), we could instead simply say only pages included in the URL mapping list are displayed on the website. I think that would be a neater mechanism.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined

microk8s.io/docs/addon-ambassador Returned 'NoneType' object, when parsing warning

Summary

For Microk8s docs, /docs/addon-ambassador is failing with error TypeError: 'NoneType' object is not callable, because the result returned by last_paragraph.contents here has not been parsed into a string. This is the value of last_paragraph.contents:

[<a href="https://www.getambassador.io/">Ambassador</a>, ' is available on the latest and 1.19+ tracks.']

Where <a href=" is not a string so it cannot access that value with last_paragraph.contents[0]

Update discourse readme

Done

Some "renovated" projects that use this package are breaking.

  • Update it with the correct parameters to be passed to make it work.

No way to navigate from Discourse to corresponding page on doc website

We now have a button to navigate from a Juju website doc to its variant on Discourse ("Help improve this document in the forum."), but not the other way around. For all Discourse docs that are actually listed on the Juju website, it would be useful to have a button to navigate back.

Handle no topic link safely

Someone added a topic without a link:
Image Pasted at 2021-1-21 13-44

This topic should be parsed and ignored if it does not match minimum criteria.

This feature should be tested to ensure to have hardening for malformed tables content.

Docs: Easier method for linking to subheadings

From canonical/juju.is#303:


It would be nice if there was a way to link to a heading in a docs page - e.g. I've just said to someone in https://chat.charmhub.io/charmhub/channels/charm-dev:

"""see the "Peer relation example" section of https://juju.is/docs/sdk/relations."""

The link already exists, as https://juju.is/docs/sdk/relations#heading--peer-relations but it is a bit hidden. To find that you have to right click the header -> inspect element -> look the HTML code for the id tag. It'd be nice if there was a visual element that let you copy that link.

Backups

Currently I believe in most cases Discourse is the only place where the documentation content lives. If Discourse gets corrupted or destroyed somehow, I think we've basically lose that content. Individual documentation editors may have downloaded documentation sets, but not in any formal way so we can't rely on this.

We should ask IS if the back up content in our Discourse installations.

Then we should think about Discourses outside our control like discuss.kubernetes.io, which contains our content. How should we back this up?

Search functionality

There is currently no way to search through the documentation. Could a search bar be implemented ?

Tutorial keys with typos cause 500s

We had an issue where a tutorial had been added with an incorrect key, "Difficulty" was set to "Difficulty level", and it caused ubuntu.com/tutorials to 500 because it couldn't find the "Difficulty" key on a recently added tutorial: https://sentry.is.canonical.com/canonical/ubuntu-com/issues/23057/?query=is%3Aunresolved

It would be good to handle that differently - maybe by excluding tutorials like that until they're fixed (not sure how you'd indicate there's a problem with the tutorial to the author though)?

We need more control over navigation HTML markup

In order to provide consistent look and feel of side navigation in documentation across sites we want to use new Vanilla side navigation component.

This component is build with specific structure of HTML with specific Vanilla class names.

DiscourseDocs for navigation simply pulls a bunch of headers and lists with links without any particular enforced structure and without any Vanilla class names.

We need DiscourseDocs to provide navigation HTML that is compatible with Vanilla side navigation pattern (having proper HTML structure and class names) or it should provide abstracted data API that returns data representation of the navigation that can be used to build the required HTML in web application view or template.

This is blocking migration of the documentation sites to proposed documentation layout with new side navigation.

Redirect discourse slugs where possible

Embedded videos

Hello team !

I am using your code to integrate our discourse content into our website, it's working great.

I have a question about embedded videos : I did some testing and noticed that within our website :

Am I doing something wrong or the YouTube link ?

Thanks in advance,
Michel

Generate anchors for headings

It is useful to be able to link to a specific section of a documentation page. I would like headings in generated docs to have anchors attached to them, and for those anchors to be exposed in the UI somehow (so I can right-click on something and "Copy Link", for example).

I think Sphinx does this well. Consider https://cloudinit.readthedocs.io/en/latest/topics/modules.html#final-message, which is a direct link to a section. The anchor name is sensible, and an inobtrusive link icon appears when you hover over the title, allowing you to easily copy the link.

Add support for emdash (—) in URLs

This character should be replaced to double dashes (--). The reason is that Discourse automatically transform -- to causing encoding errors in Gunicorn.

[Feature] Configure URL mappings in Python code rather than in Discourse

For Vanilla docs we need the discourse content to be aligned with docs we have in the code. We also don't need the navigation handled and maintained in Discourse.

It would be good if we could configure URL mappings (path to discourse post) in Python rather than having to maintain empty page on discourse just to maintain a "Navigation" table there.

It seems that if we could pass URL mapping directly to discourse module it could also directly fetch the required content without the need to fetch the index topic and parse Navigation out of it.

Some "pseudo-code" suggestion to what I mean:

discourse_docs = Docs(
    parser=DocParser(
        api=DiscourseAPI(
            base_url="https://discourse.ubuntu.com/", session=session
        ),
        # instead of index_topic pass the URL mapping
        # mapping could be full discourse URLs, just paths, or maybe just post IDs
        url_mapping = {
           "pattern/notification": "/t/12345",
           "pattern/code-snippet": "/t/54321",
        },
        url_prefix="/design",
    ),
    document_template="/_layouts/docs_discourse.html",
    url_prefix="/design",
)

Images are optimized without the possiblity to see the full size version

On discourse, images are first displayed in their smaller, optimized, version, and the full version can be shown by clicking on them.

This module seems to do the same optimization but does not provide a way to enlarge the image by clicking on it. This results in blurry images as can be seen in this page.

For reference here's what discourse look like

Expected behavior
The image should either be fetched in full resolution, or the page should provide a way to click on the optimized image to load the full resolution

Current behavior
Only the smaller optimized image is available

[Feature] More control over the URL routes the module handles

Currently discourse module is configured by passing a urlPrefix to it and ALL URLs under this prefix are served from discourse.

For Vanilla docs where we have part of docs served from discourse and part from Flask webapp, we'd like to have more control on what URLs are handled by discourse.

In our case we'd like to keep current URL structure of /docs/patterns/notification being served by Flask, and only a design guidelines subpage /docs/patterns/notification/design to be fetched from discourse.

Our current workaround is to have a /design prefix for discourse, so having URLs of /design/patterns/notification alongside /docs/patterns/notification, but this is not ideal.

Would it be possible to have more control over which routes are handled by discourse module and not assign whole URL prefix to it?

It would be nice if we can make it in a way that just works with Flask routes. Something like:

@app.route("/docs/<path:pattern_path>/design")
def get_design_guidelines(pattern_path):
    # get content from discourse based on the path

[content list] create discourse pages list

Should have the content of the metadata table, perhaps without the ‘active’ column.

topic path type tags publish_date language subtitle
Simplifying AI/ML adoption in telco with modern operations practices /engage/ai-ml-telco-modern-operations webinar AI/ML, telecommunications 2021-02-17   Leveraging open source software for optimal TCO
Enterprise Kubernetes: A buyer’s guide /engage/enterprise-kubernetes-guide whitepaper Kubernetes, K8s, Cloud 2021-02-16   Find the right Kubernetes for your business

Ideally this would output a page for docs as well.

add `noindex` option to engage pages

We have a use case for engage pages that we don't want to show up in search results, or be listed on the engage index page of ubuntu.com, could we add something like a noindex metadata field to those posts, and handle it in ubuntu.com?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.