posthog / housewatch

Open source tool for monitoring and managing ClickHouse clusters

📈 Open source tool for monitoring and managing ClickHouse clusters

• Get an overview of cluster load and performance
• Drill down into your queries and understand the load they put on your cluster
• Search through logs and errors
• Monitor and kill running queries with the click of a button
• Get stats on your disk usage per node, and understand how much disk space tables, columns, and parts take up
• Run your own queries straight from the interface to further dig into performance and cluster issues
• Setup operations to run in the background with automatic rollbacks for failures

💻 Deploy

To deploy HouseWatch, clone this repo and then run the following, substituting the environment variables for the relevant values of one of your ClickHouse instances:

SITE_ADDRESS=<SITE_ADDRESS> \
CLICKHOUSE_HOST=localhost \
CLICKHOUSE_CLUSTER=mycluster \
CLICKHOUSE_USER=default \
CLICKHOUSE_PASSWORD=xxxxxxxxxxx \
docker compose -f docker-compose.yml up

SITE_ADDRESS here is the address that the UI will be running on. It can be a domain name or simply a port like :80.

After running the above, the UI will be running on the address you specified. This will be something like http://localhost if you used :80 for your SITE_ADDRESS above. I would think twice about exposing this to the internet, as it is not currently secured in any way.

Read more

The following are the supported environment variables for configuring your HouseWatch deployment:

• CLICKHOUSE_HOST: Required - hostname of the instance to connect to.
• CLICKHOUSE_USER: Required - username to access ClickHouse. Can be a read-only user, but in that case not all features will work.
• CLICKHOUSE_PASSWORD: Required - password for the specified user.
• CLICKHOUSE_DATABASE: Optional - database to connect to by default.
• CLICKHOUSE_CLUSTER: Optional - cluster name, to analyze data from the whole cluster.
• CLICKHOUSE_SECURE: Optional - see clickhouse-driver docs for more information
• CLICKHOUSE_VERIFY: Optional - see clickhouse-driver docs for more information
• CLICKHOUSE_CA: Optional - see clickhouse-driver docs for more information
• OPENAI_API_KEY: Optional - enables the experimental "AI Tools" page, which currently features a natural language query editor
• OPENAI_MODEL: Optional - a valid OpenAI model (e.g. gpt-3.5-turbo, gpt-4) that you have access to with the key above to be used for the AI features

🏡 Running locally

To run HouseWatch locally along with a local ClickHouse instance, execute:

docker compose -f docker-compose.dev.yml up -d

then go to http://localhost:8080

💡 Motivation

At PostHog we manage a few large ClickHouse clusters and found ourselves in need of a tool to monitor and manage these more easily.

ClickHouse is fantastic at introspection, providing a lot of metadata about the system in its system tables so that it can be easily queried. However, knowing exactly how to query and parse the available information can be a difficult task. Over the years at PostHog, we've developed great intuition for how to debug ClickHouse issues using ClickHouse, and HouseWatch is the compilation of this knowledge into a tool.

Beyond monitoring, we also built internal systems and processes for managing the clusters that spanned various platforms. We would use Grafana to look at metrics, SSH into nodes for running operations and using specialized tooling, query via Metabase to dig deeper into the data in the system tables and create dashboards, and then a combination of tools baked into the PostHog product for further debugging and streamlined operations such as our async migrations tool, and internal views for listing queries and analyzing their performance.

As a result, we felt it was appropriate to have these tools live in one place. Ultimately, our vision for HouseWatch is that it can both serve the purpose of a pganalyze for the ClickHouse ecosystem, while also including tooling for taking action on insights derived from the analysis.

🏗️ Status of the project

HouseWatch is in its early days and we have a lot more features in mind that we'd like to build into it going forward. The code could also use some cleaning up :) As of right now, it is considered Beta software and you should exercise caution when using it in production.

One potential approach is to connect HouseWatch to ClickHouse using a read-only user. In this case, the cluster management features will not work (e.g. operations, query editor), but the analysis toolset will function normally.

HouseWatch was created and is maintained by PostHog and yakkomajuri.

ℹ️ Contributing

Contributions are certainly welcome! However, if you'd like to build a new feature, please open up an issue first.

⭐ Features

Query performance

Schema stats

Query benchmarking

Logs

Query editor

Disk usage

Errors

Operations

🗒️ To-do list

A public list of things we intend to do with HouseWatch in the near future.

See list

Features

• System issues tab
• EXPLAIN visualizer
• Multiple instance support
• Stats on page cache hit percentage
• Make operations resilient to Celery going down (as we do in PostHog with async migrations)
• Read-only mode
• Button to force refresh running queries list
• Logs pagination
• Allow copying example queries
• Configurable time ranges
• Whole cluster schema stats
• More operation controls: view, delete, edit, re-run, display errors

Developer experience

• Configure instance from UI
• Publish a Docker image
• Development docker-compose.yml with baked in ClickHouse

Cleanup

• Extract README images out of repo
• Make banner subtitle work on dark mode
• Fetch data independently on the query analyzer
• Breakpoint for logs search
• Run Django "production server"
• Write tests :)
• Query editor pipe all errors to client
• Abstraction to load data from API as JSON

housewatch's Issues

Monitor whole cluster

Hello, I wanted to ask if I can use HouseWatch to monitor all the cluster nodes or only the one connected to it?

Cant start docker compose: => ERROR [web 3/3] COPY build/ build/

The default docker-compose.yml has an incorrect ref to "clickhouse" service. Should this be removed? I have a clickhouse cluster deployed in AWS and I am providing the hostname via the Env vars:

SITE_ADDRESS=:80 \
CLICKHOUSE_HOST=<AWS ALB hostname>\
CLICKHOUSE_CLUSTER=mycluster \
CLICKHOUSE_USER= <someusername> \
CLICKHOUSE_PASSWORD=xxxxxxxxxxx \
docker compose -f docker-compose.yml up

After removing the reference of "clickhouse" service under the depends_on section, the docker instances fails to start with the following error:

 => CANCELED [worker 1/8] FROM docker.io/library/python:3.10@sha256:1a8dcc07368065c2b285b24236a98e80355d6de7f685b416407aafb6dd32529f                                                                                 0.5s
 => => resolve docker.io/library/python:3.10@sha256:1a8dcc07368065c2b285b24236a98e80355d6de7f685b416407aafb6dd32529f                                                                                                 0.0s
 => => sha256:aa51dc5b3e4b04c28168c424bb88a7329fa5ddea735617e70aca8e276bed838a 7.53kB / 7.53kB                                                                                                                       0.0s
 => => sha256:00046d1e755ea94fa55a700ca9a10597e4fac7c47be19d970a359b0267a51fbf 0B / 24.03MB                                                                                                                          0.4s
 => => sha256:9f13f5a53d118643c1f1ff294867c09f224d00edca21f56caa71c2321f8ca004 0B / 64.11MB                                                                                                                          0.4s
 => => sha256:1a8dcc07368065c2b285b24236a98e80355d6de7f685b416407aafb6dd32529f 1.65kB / 1.65kB                                                                                                                       0.0s
 => => sha256:17b967fa748a2929eb2682018a9d0e335aeebff3963469e901b67544bd01be49 2.01kB / 2.01kB                                                                                                                       0.0s
 => => sha256:012c0b3e998c1a0c0bedcf712eaaafb188580529dd026a04aa1ce13fdb39e42b 0B / 49.56MB                                                                                                                          0.4s
 => [worker internal] load build context                                                                                                                                                                             0.0s
 => => transferring context: 3.49kB                                                                                                                                                                                  0.0s
 => [app internal] load build context                                                                                                                                                                                0.0s
 => => transferring context: 3.49kB                                                                                                                                                                                  0.0s
 => CACHED [web 1/3] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a                                                                                    0.0s
 => [web internal] load build context                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                      0.0s
 => [web 2/3] WORKDIR /frontend                                                                                                                                                                                      0.0s
 => ERROR [web 3/3] COPY build/ build/                                                                                                                                                                               0.0s
------
 > [web 3/3] COPY build/ build/:
------
failed to solve: failed to compute cache key: failed to calculate checksum of ref b31f82cc-6f48-4507-bec3-385667894ed8::e8fy1zrfjzzwwtfpwoj97s5ac: "/build": not found

Am I doing something wrong? The error message indicates that there is an issue with the COPY command in the Dockerfile. What would be the next steps to resolve this?

schema stats page error on loading data

We are using ClickHouse server 23.12.2, maybe there was an update in field names of system.tables leading to frontend code getting wrong fields

Frontend is reading name field, but query return table field instead:

I recommend to add new fields on query in housewatch/clickhouse/queries/sql.py, it won't break previous HouseWatch versions

from this:

AVAILABLE_TABLES_SQL = """
SELECT database, table
FROM system.tables
WHERE database NOT ILIKE 'information_schema'
"""

to this:

AVAILABLE_TABLES_SQL = """
SELECT database, table, table as name, total_bytes, total_bytes as readable_bytes, total_rows, engine, partition_key
FROM system.tables
WHERE database NOT ILIKE 'information_schema'
"""

if that okay I will submit a pull request

Logo in dark mode

Hi, I'm participating in an event for open source contributions. As I couldn't fix your code, I may be creating a version of your banner for dark mode, as the current one doesn't fit well. The only thing I would need was the original font.

Not working. The script is not reading the environment variables.

The script is not reading the environment variables that I supply in Terminal. It doesn't even take them if I hard code them in docker compose file. I get error as well.

housewatch-redis-1   | 1:C 23 Jun 2023 10:18:42.468 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
housewatch-redis-1   | 1:C 23 Jun 2023 10:18:42.472 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
housewatch-redis-1   | 1:C 23 Jun 2023 10:18:42.472 # Configuration loaded
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.477 * Increased maximum number of open files to 10032 (it was originally set to 1024).
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.479 * monotonic clock: POSIX clock_gettime
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.484 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.484 * Running mode=standalone, port=6379.
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.485 # Server initialized
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.485 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.487 * Loading RDB produced by version 6.2.7
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.491 * RDB age 33 seconds
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.491 * RDB memory usage when created 0.77 Mb
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.492 # Done loading RDB, keys loaded: 0, keys expired: 0.
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.492 * DB loaded from disk: 0.006 seconds
housewatch-redis-1   | 1:M 23 Jun 2023 10:18:42.493 * Ready to accept connections
housewatch-web-1     |  INFO  Accepting connections at http://localhost:3000
housewatch-app-1     | Failed to connect to :9440
housewatch-app-1     | Traceback (most recent call last):
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 342, in connect
housewatch-app-1     |     return self._init_connection(host, port)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 306, in _init_connection
housewatch-app-1     |     self.socket = self._create_socket(host, port)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 257, in _create_socket
housewatch-app-1     |     for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
housewatch-app-1     |   File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
housewatch-app-1     |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
housewatch-app-1     | socket.gaierror: [Errno -2] Name or service not known
housewatch-app-1     | Traceback (most recent call last):
housewatch-app-1     |   File "/code/manage.py", line 22, in <module>
housewatch-app-1     |     main()
housewatch-app-1     |   File "/code/manage.py", line 18, in main
housewatch-app-1     |     execute_from_command_line(sys.argv)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
housewatch-app-1     |     utility.execute()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
housewatch-app-1     |     self.fetch_command(subcommand).run_from_argv(self.argv)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
housewatch-app-1     |     self.execute(*args, **cmd_options)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
housewatch-app-1     |     output = self.handle(*args, **options)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 96, in wrapped
housewatch-app-1     |     res = handle_func(*args, **kwargs)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 97, in handle
housewatch-app-1     |     self.check(databases=[database])
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 475, in check
housewatch-app-1     |     all_issues = checks.run_checks(
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/registry.py", line 88, in run_checks
housewatch-app-1     |     new_errors = check(app_configs=app_configs, databases=databases)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/urls.py", line 14, in check_url_config
housewatch-app-1     |     return check_resolver(resolver)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/urls.py", line 24, in check_resolver
housewatch-app-1     |     return check_method()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 494, in check
housewatch-app-1     |     for pattern in self.url_patterns:
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
housewatch-app-1     |     res = instance.__dict__[self.name] = self.func(instance)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 715, in url_patterns
housewatch-app-1     |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
housewatch-app-1     |     res = instance.__dict__[self.name] = self.func(instance)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
housewatch-app-1     |     return import_module(self.urlconf_name)
housewatch-app-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
housewatch-app-1     |     return _bootstrap._gcd_import(name[level:], package, level)
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
housewatch-app-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
housewatch-app-1     |   File "/code/housewatch/urls.py", line 6, in <module>
housewatch-app-1     |     from housewatch.api.analyze import AnalyzeViewset
housewatch-app-1     |   File "/code/housewatch/api/analyze.py", line 5, in <module>
housewatch-app-1     |     from housewatch.clickhouse.client import run_query, ch_host, existing_system_tables
housewatch-app-1     |   File "/code/housewatch/clickhouse/client.py", line 59, in <module>
housewatch-app-1     |     existing_system_tables = [row["name"] for row in run_query(EXISTING_TABLES_SQL, use_cache=False)]
housewatch-app-1     |   File "/code/housewatch/clickhouse/client.py", line 46, in run_query
housewatch-app-1     |     result = client.execute(final_query, settings=settings, with_column_types=True, query_id=query_id)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 345, in execute
housewatch-app-1     |     with self.disconnect_on_error(query, settings):
housewatch-app-1     |   File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
housewatch-app-1     |     return next(self.gen)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 289, in disconnect_on_error
housewatch-app-1     |     self.establish_connection(settings)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 276, in establish_connection
housewatch-app-1     |     self.connection.force_connect()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 235, in force_connect
housewatch-app-1     |     self.connect()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 363, in connect
housewatch-app-1     |     raise err
housewatch-app-1     | clickhouse_driver.errors.NetworkError: Code: 210. Name or service not known (:9440)
housewatch-worker-1  | Failed to connect to :9440
housewatch-worker-1  | Traceback (most recent call last):
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 342, in connect
housewatch-worker-1  |     return self._init_connection(host, port)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 306, in _init_connection
housewatch-worker-1  |     self.socket = self._create_socket(host, port)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 257, in _create_socket
housewatch-worker-1  |     for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
housewatch-worker-1  |   File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
housewatch-worker-1  |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
housewatch-worker-1  | socket.gaierror: [Errno -2] Name or service not known
housewatch-worker-1  | Traceback (most recent call last):
housewatch-worker-1  |   File "/usr/local/bin/celery", line 8, in <module>
housewatch-worker-1  |     sys.exit(main())
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/__main__.py", line 15, in main
housewatch-worker-1  |     sys.exit(_main())
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/bin/celery.py", line 217, in main
housewatch-worker-1  |     return celery(auto_envvar_prefix="CELERY")
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
housewatch-worker-1  |     return self.main(*args, **kwargs)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
housewatch-worker-1  |     rv = self.invoke(ctx)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
housewatch-worker-1  |     return _process_result(sub_ctx.command.invoke(sub_ctx))
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
housewatch-worker-1  |     return ctx.invoke(self.callback, **ctx.params)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
housewatch-worker-1  |     return __callback(*args, **kwargs)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
housewatch-worker-1  |     return f(get_current_context(), *args, **kwargs)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/bin/base.py", line 134, in caller
housewatch-worker-1  |     return f(ctx, *args, **kwargs)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/bin/worker.py", line 343, in worker
housewatch-worker-1  |     worker = app.Worker(
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/worker/worker.py", line 94, in __init__
housewatch-worker-1  |     self.app.loader.init_worker()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/loaders/base.py", line 111, in init_worker
housewatch-worker-1  |     self.import_default_modules()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/loaders/base.py", line 105, in import_default_modules
housewatch-worker-1  |     raise response
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/utils/dispatch/signal.py", line 276, in send
housewatch-worker-1  |     response = receiver(signal=self, sender=sender, **named)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/fixups/django.py", line 82, in on_import_modules
housewatch-worker-1  |     self.worker_fixup.validate_models()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/fixups/django.py", line 121, in validate_models
housewatch-worker-1  |     run_checks()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/registry.py", line 88, in run_checks
housewatch-worker-1  |     new_errors = check(app_configs=app_configs, databases=databases)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/urls.py", line 14, in check_url_config
housewatch-worker-1  |     return check_resolver(resolver)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/urls.py", line 24, in check_resolver
housewatch-worker-1  |     return check_method()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 494, in check
housewatch-worker-1  |     for pattern in self.url_patterns:
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
housewatch-worker-1  |     res = instance.__dict__[self.name] = self.func(instance)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 715, in url_patterns
housewatch-worker-1  |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
housewatch-worker-1  |     res = instance.__dict__[self.name] = self.func(instance)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
housewatch-worker-1  |     return import_module(self.urlconf_name)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
housewatch-worker-1  |     return _bootstrap._gcd_import(name[level:], package, level)
housewatch-worker-1  |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
housewatch-worker-1  |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
housewatch-worker-1  |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
housewatch-worker-1  |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
housewatch-worker-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
housewatch-worker-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
housewatch-worker-1  |   File "/code/housewatch/urls.py", line 6, in <module>
housewatch-worker-1  |     from housewatch.api.analyze import AnalyzeViewset
housewatch-worker-1  |   File "/code/housewatch/api/analyze.py", line 5, in <module>
housewatch-worker-1  |     from housewatch.clickhouse.client import run_query, ch_host, existing_system_tables
housewatch-worker-1  |   File "/code/housewatch/clickhouse/client.py", line 59, in <module>
housewatch-worker-1  |     existing_system_tables = [row["name"] for row in run_query(EXISTING_TABLES_SQL, use_cache=False)]
housewatch-worker-1  |   File "/code/housewatch/clickhouse/client.py", line 46, in run_query
housewatch-worker-1  |     result = client.execute(final_query, settings=settings, with_column_types=True, query_id=query_id)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 345, in execute
housewatch-worker-1  |     with self.disconnect_on_error(query, settings):
housewatch-worker-1  |   File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
housewatch-worker-1  |     return next(self.gen)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 289, in disconnect_on_error
housewatch-worker-1  |     self.establish_connection(settings)
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 276, in establish_connection
housewatch-worker-1  |     self.connection.force_connect()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 235, in force_connect
housewatch-worker-1  |     self.connect()
housewatch-worker-1  |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 363, in connect
housewatch-worker-1  |     raise err
housewatch-worker-1  | clickhouse_driver.errors.NetworkError: Code: 210. Name or service not known (:9440)
housewatch-worker-1 exited with code 1
housewatch-app-1     | Performing system checks...
housewatch-app-1     | 
housewatch-app-1     | Failed to connect to :9440
housewatch-app-1     | Traceback (most recent call last):
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 342, in connect
housewatch-app-1     |     return self._init_connection(host, port)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 306, in _init_connection
housewatch-app-1     |     self.socket = self._create_socket(host, port)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 257, in _create_socket
housewatch-app-1     |     for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
housewatch-app-1     |   File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
housewatch-app-1     |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
housewatch-app-1     | socket.gaierror: [Errno -2] Name or service not known
housewatch-app-1     | Failed to connect to :9440
housewatch-app-1     | Traceback (most recent call last):
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 342, in connect
housewatch-app-1     |     return self._init_connection(host, port)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 306, in _init_connection
housewatch-app-1     |     self.socket = self._create_socket(host, port)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 257, in _create_socket
housewatch-app-1     |     for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
housewatch-app-1     |   File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
housewatch-app-1     |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
housewatch-app-1     | socket.gaierror: [Errno -2] Name or service not known
housewatch-app-1     | Exception in thread django-main-thread:
housewatch-app-1     | Traceback (most recent call last):
housewatch-app-1     |   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
housewatch-app-1     |     self.run()
housewatch-app-1     |   File "/usr/local/lib/python3.10/threading.py", line 953, in run
housewatch-app-1     |     self._target(*self._args, **self._kwargs)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 64, in wrapper
housewatch-app-1     |     fn(*args, **kwargs)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 134, in inner_run
housewatch-app-1     |     self.check(display_num_errors=True)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 475, in check
housewatch-app-1     |     all_issues = checks.run_checks(
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/registry.py", line 88, in run_checks
housewatch-app-1     |     new_errors = check(app_configs=app_configs, databases=databases)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/urls.py", line 14, in check_url_config
housewatch-app-1     |     return check_resolver(resolver)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/core/checks/urls.py", line 24, in check_resolver
housewatch-app-1     |     return check_method()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 494, in check
housewatch-app-1     |     for pattern in self.url_patterns:
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
housewatch-app-1     |     res = instance.__dict__[self.name] = self.func(instance)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 715, in url_patterns
housewatch-app-1     |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
housewatch-app-1     |     res = instance.__dict__[self.name] = self.func(instance)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
housewatch-app-1     |     return import_module(self.urlconf_name)
housewatch-app-1     |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
housewatch-app-1     |     return _bootstrap._gcd_import(name[level:], package, level)
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
housewatch-app-1     |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
housewatch-app-1     |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
housewatch-app-1     |   File "/code/housewatch/urls.py", line 6, in <module>
housewatch-app-1     |     from housewatch.api.analyze import AnalyzeViewset
housewatch-app-1     |   File "/code/housewatch/api/analyze.py", line 5, in <module>
housewatch-app-1     |     from housewatch.clickhouse.client import run_query, ch_host, existing_system_tables
housewatch-app-1     |   File "/code/housewatch/clickhouse/client.py", line 59, in <module>
housewatch-app-1     |     existing_system_tables = [row["name"] for row in run_query(EXISTING_TABLES_SQL, use_cache=False)]
housewatch-app-1     |   File "/code/housewatch/clickhouse/client.py", line 46, in run_query
housewatch-app-1     |     result = client.execute(final_query, settings=settings, with_column_types=True, query_id=query_id)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 345, in execute
housewatch-app-1     |     with self.disconnect_on_error(query, settings):
housewatch-app-1     |   File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
housewatch-app-1     |     return next(self.gen)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 289, in disconnect_on_error
housewatch-app-1     |     self.establish_connection(settings)
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/client.py", line 276, in establish_connection
housewatch-app-1     |     self.connection.force_connect()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 235, in force_connect
housewatch-app-1     |     self.connect()
housewatch-app-1     |   File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 363, in connect
housewatch-app-1     |     raise err
housewatch-app-1     | clickhouse_driver.errors.NetworkError: Code: 210. Name or service not known (:9440)
^CGracefully stopping... (press Ctrl+C again to force)

Guide for running with clickhouse docker

I think a guide would be nice -- for some reason I can't get this to connect to my local clickhouse instance running on docker.

How I'm starting docker instance (on macOS m1 chip):

docker run -d -p 8123:8123 -p 9000:9000 \
    -v "$(pwd)/ch_data":/var/lib/clickhouse/ \
    -v "$(pwd)/ch_logs":/var/log/clickhouse-server/ \
    -v "$(pwd)/config.d":/etc/clickhouse-server/config.d/ \
    --name local-clickhouse --ulimit nofile=262144:262144 \
    -e CLICKHOUSE_DB=my_database -e CLICKHOUSE_USER=username -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 -e CLICKHOUSE_PASSWORD=1234 \
    clickhouse/clickhouse-server

How I start housewatch:

CLICKHOUSE_HOST=localhost \
CLICKHOUSE_CLUSTER=local-clickhouse \
CLICKHOUSE_USER=username \
CLICKHOUSE_PASSWORD=1234 \
CLICKHOUSE_SECURE=false \
docker compose -f docker-compose.yml up

Resulting errors

housewatch-app-1 | Failed to connect to localhost:9000

housewatch-worker-1 | Failed to connect to localhost:9000

housewatch-web-1 |
housewatch-web-1 | Proxy error: Could not proxy request /api/analyze/hostname from localhost:3000 to http://localhost:8000.
housewatch-web-1 | See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNREFUSED).
housewatch-web-1 |

Would love any insights.

analyze/logs returns 500

For k8s
https://github.com/PostHog/HouseWatch/blob/5811b3f717e6faca78c1f3bd7db454324d0a4767/housewatch/clickhouse/queries/sql.py#L6C112-L6C112

1. According to the documentation there should be "if ch_cluster else "system.text_log"
   https://clickhouse.com/docs/ru/operations/system-tables/text_log system.text_log or returns "POST /api/analyze/logs HTTP/1.1" 500 86
2. If I use name CLICKHOUSE_CLUSTER (ch_cluster), it shows 500 other tabs /api/analyze/

What is the repo url and chart name to install via helm chart

error - Serve the files from /frontend/build, don't run this container directly

Hi

Tried to deploy HouseWatch and receiving this from the web container, anything I am doing wrong?

pls add k8s deploy manifest

recommend use https://github.com/kubernetes/kompose to transform compose file to Kubernetes resources manifest

helm chart packaging/nginx configuration

index.html requests resources from /webapp/assets/... but assets is a sibling of index.html, not stored in a /webapp/ subfolder therefore assets are not found.

Either index.html must be changed to not use /webapp/ subfolder or nginx must be configured to remap /webapp/ to top level .

Frontend always request to localhost:8000

I found all request from frontend always come to localhost:8000, which is not true if you host the webserver on server then woking on local machine browser, should I make an improvement by adding server host as an env?

After docker-compose up and i go to the site, method /api/analyze/query_graphs return 500

Hi! I decided to try your tool, but I can't use it

In localhost:3000 i see erros:

backend: http api method /api/analyze/query_graphs return 500 with body {"type":"server_error","code":"error","detail":"A server error occurred.","attr":null}

frontend:

logs:

housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 7 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/css/1.f7ef86df.chunk.css
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/css/main.6e4c166f.chunk.css
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 5 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 5 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/js/1.fb96ed73.chunk.js
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/js/main.a91ea030.chunk.js
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 2 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 2 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/css/main.6e4c166f.chunk.css.map
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/css/1.f7ef86df.chunk.css.map
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 1 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 1 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/js/1.fb96ed73.chunk.js.map
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/js/main.a91ea030.chunk.js.map
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 1 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 2 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /static/media/HindSiliguri-Medium.41fd138d.ttf
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 1 ms
housewatch-app-1 | [14/Jul/2023 12:45:08] "GET /api/analyze/hostname HTTP/1.1" 200 29
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /favicon.ico
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 2 ms
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 GET /manifest.json
housewatch-web-1 | HTTP 7/14/2023 12:45:08 PM 172.18.0.1 Returned 304 in 1 ms
housewatch-app-1 | [14/Jul/2023 12:45:08] "GET /api/analyze/query_graphs HTTP/1.1" 500 86

I use environment variables: CLICKHOUSE_SECURE=False, CLICKHOUSE_HOST , CLICKHOUSE_CLUSTER, CLICKHOUSE_USER and CLICKHOUSE_PASSWORD

With respect, Andrei

Why use port 9440?

Why not use port 8123 or 9000?

housewatch-app-1 | Failed to connect to localhost:9440
housewatch-app-1 | Traceback (most recent call last):
housewatch-app-1 | File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 342, in connect
housewatch-app-1 | return self._init_connection(host, port)
housewatch-app-1 | File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 306, in _init_connection
housewatch-app-1 | self.socket = self._create_socket(host, port)
housewatch-app-1 | File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 278, in _create_socket
housewatch-app-1 | raise err
housewatch-app-1 | File "/usr/local/lib/python3.10/site-packages/clickhouse_driver/connection.py", line 269, in _create_socket
housewatch-app-1 | sock.connect(sa)
housewatch-app-1 | File "/usr/local/lib/python3.10/ssl.py", line 1375, in connect
housewatch-app-1 | self._real_connect(addr, False)
housewatch-app-1 | File "/usr/local/lib/python3.10/ssl.py", line 1362, in _real_connect