New anti-bot "feature" about kleinanzeigen-bot

[elgarfo]

as i was unable to find clear instructions on what "working profile" meant and i suspect others to feel the same: this is the way i got it to work.

⚠️: i am using docker on debian 12 in wsl2 on windows with vcxsrv as my display server. YMMV. that being said, i can imagine this is universally working for the docker-version.

setup config

as mentioned above, i modified the chromium options in config.yaml:

browser:
...
  use_private_window: false
  user_data_dir: '/mnt/data/userdata'
  profile_name: 'profile'

this provides a persistent storage for the docker-based chromium. i chose /mnt/data/userdir as that was already mounted as a volume in the example commands.

prepare persistent profile

to prepare the profile. you need to login manually to your kleinanzeigen.de account. as the bot gives very little time and often triggers the captcha and blocked message, i recommend manually running chromium for the first login. this will open a chromium instance with the configured persistent profile data. just surf to https://kleinanzeigen.de/, manually login and close the browser afterwards.

docker run --rm -it --shm-size=256m \
  -e DISPLAY=$X11_DISPLAY \
  -v ${PWD}:/mnt/data \
  --entrypoint /usr/bin/chromium \
  ghcr.io/second-hand-friends/kleinanzeigen-bot \
  --user-data-dir=/mnt/data/userdata/ \
  --profile-directory=profile \
  --disable-crash-reporter \
  --no-first-run \
  --no-service-autorun \
  --no-sandbox

thats it. your profile is now holding session-data for your kleinanzeigen.de account and will work with commands like docker run --rm -it --shm-size=256m -e DISPLAY=$X11_DISPLAY -v ${PWD}:/mnt/data ghcr.io/second-hand-friends/kleinanzeigen-bot download to download all your ads.

using multiple accounts

if you want to use multiple kleinanzeigen.de accounts, you can repeat the preparation step for each individual account and just change the --profile-directory parameter together with a matching config.yaml

i did not use DreckSofts fixed version with the undetected-chromedriver, as i did not want to build from source. hopefully the login-fix gets eventually merged. as of now i'm fine with this workaround.

[ghoeffner]

Getting this too, seems related to a new type of login form.

[sebthom]

We may have to look into using https://pypi.org/project/undetected-chromedriver/

[DreckSoft]

A workaround is to create a profile in Chromium per Kleinanzeigen account and use that in a non-private window. Then you only have to login once. It's more convenient anyways, saves the captcha at the beginning.

[ilyaZar]

@DreckSoft Would you mind giving a bit more details as a preliminary workaround for this ? I get blogged to an extent that it almost always ends up on this page, sometimes after a few restarts worked before but now it seems not to work at all (at least for me) ....

1. In the config.yaml I am setting
   

2. So here is the screenshot from chromium

[apeltzer]

A workaround is to create a profile in Chromium per Kleinanzeigen account and use that in a non-private window. Then you only have to login once. It's more convenient anyways, saves the captcha at the beginning.

Couldnt get this to work myself either. It always takes the default profile, even when setting the entry to false and the profile path properly.

[jonmert]

I'm experiencing the same problem. Even after completing the captcha manually I am getting blocked.

[DreckSoft]

Launching chromedriver with the correct profile works for me but I'm no longer logged in. If I start Chromium with that profile I am. So really strange.

[DreckSoft]

It seems it is getting worse. When I start Chromium, everything works and I'm logged in. With Chromedriver I currently ALWAYS getting the anti-bot thing. So no publishing possible at all.

[DreckSoft]

We may have to look into using https://pypi.org/project/undetected-chromedriver/

That seems to require more adjustments than just importing undetected_chromedriver :(

[mcsurfi]

The problem still exists....

[KeksTheAllmighty]

Having the same issue with google-chrome on ubuntu... tried using some google-chrome crx extension files but nothing works :'(

[DreckSoft]

The following works at least with my patch (haven't tried without):

• You MUST have a working setup to use an existing Chromium profile. See instructions on the main page.
• Open Chromium manually and log in.
• Go to your ads page
• Close Chromium
• Start the app. It will go to the login page and submit user/pw but will immediately go back to the login page.
• When that is the case enter https://kleinanzeigen.de into the address bar (better have that copied to the clipboard before and just paste)
• If the timing was right (!), it will detect that you are already logged in. If not, try again.

[DreckSoft]

My current version fixes the login check. So you just need a working Chromium profile and need to be logged in.

I can't do another pull request as I haven't synced to the latest changes with the dependencies update.

https://github.com/DreckSoft/kleinanzeigen-bot

No binaries are provided you will have to use the source code version with pdm.

[ilyaZar]

@elgarfo this explanation is perfect thanks, I also struggled with the notion of "chrome profile" and a more elaborated explanation for the workaround greatly helps 👍

[RomanRadkoMobile]

Hi guys,
Dose somebody faced this "Chromium has locked the profile issue..." ?

[DenisLanz]

After wasting couple of hours trying to get @elgarfo 's solution with docker to work on macOS with Docker and XQuartz I gave up… just to figure out a way easier solution to use an existing profile and it's sessions on macOS.

Start chrome with the remote debugging port mapped to 9222 in one terminal session:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

Log In to eBay Kleinanzeigen and keep the Window Open.

and execute kleinanzeigen-bot through pdm pdm run app publish --config=<config file> in another terminal session with the following settings:

browser:
  arguments:
    - --disable-dev-shm-usage
    - --no-sandbox
    - --remote-debugging-port=9222
  binary_location: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
  use_private_window: false
  user_data_dir: '~/Library/Application Support/Google/Chrome'
  profile_name: 'Default'

It might be necessary to change the ~ to the absolute path to the User Profile - just changed it for the sake of sharing it here.

[lamyergeier]

@DenisLanz I modified your instructions for Linux, but I was unsuccessful in skipping the issue posted by the OP.

[DenisLanz]

Try adding - --profile-directory='Default'to the selenium arguments. I had this in but wasn't sure if it is the same as the Kleinanzeigen bot profile_name settings. Published all my ads for now so I couldn't test if it's necessary or not. The timeout error from your other issue indicates that the profile is not loaded correctly and you're not reusing the exisiting session. If you don't see an [INFO] Already logged in as [<username>]. Skipping login. in the pdm log and the ads are created in the same window you're logged in before there is an issue with the profile.

[TylonHH]

Try adding

you mean like this?

  # https://peter.sh/experiments/chromium-command-line-switches/
  arguments:
    # https://stackoverflow.com/a/50725918/5116073
    - --disable-dev-shm-usage
    - --no-sandbox
    - --profile-directory='Default'
    # --headless
    # --start-maximized
  binary_location: 'C:\Program Files\Google\Chrome\Application\chrome.exe' # path to custom browser executable, if not specified will be looked up on PATH
  extensions: [] # a list of .crx extension files to be loaded
  use_private_window: false
  user_data_dir: '' # see https://github.com/chromium/chromium/blob/main/docs/user_data_dir.md
  profile_name: ''

[TylonHH]

this

I get this with this config. It still generate a new session:

2024-01-15 15:29:45,277 [INFO] App version: 2024+8389123
2024-01-15 15:29:45,277 [INFO] Python version: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
2024-01-15 15:29:45,278 [WARNING] You provided no ads selector. Defaulting to "new".
2024-01-15 15:29:45,287 [INFO] Loading config from [C:\TEMP\kleinanzeigen-bot\config.yaml]...
2024-01-15 15:29:45,328 [INFO]  -> found 145 categories
2024-01-15 15:29:45,329 [INFO] Creating WebDriver session...
2024-01-15 15:29:45,367 [INFO] Using Browser: GOOGLE-CHROME 120.0.6099.217 [C:\Program Files\Google\Chrome\Application\chrome.exe]
2024-01-15 15:29:47,130 [INFO]  -> Browser User Data Dir: C:\Users\Rene\AppData\Local\Google\Chrome\User Data\Default
2024-01-15 15:29:47,131 [INFO]  -> Browser Profile Name: Default
2024-01-15 15:29:47,132 [INFO]  -> Custom chrome argument: --disable-dev-shm-usage
2024-01-15 15:29:47,133 [INFO]  -> Custom chrome argument: --no-sandbox
2024-01-15 15:29:47,133 [INFO]  -> Custom chrome argument: --profile-directory='Default'
2024-01-15 15:29:47,133 [INFO]  -> Chrome binary location: C:\Program Files\Google\Chrome\Application\chrome.exe
2024-01-15 15:29:48,494 [INFO] New WebDriver session is: 8716a3ba62a0f315f5840277df7b1dcf http://localhost:62614
2024-01-15 15:29:48,495 [INFO] Checking if already logged in
2024-01-15 15:29:49,874 [INFO]  ... pausing for 2000 ms ...
2024-01-15 15:29:56,978 [INFO] Logging in as [tylonhh............

[DenisLanz]

You need to check how you can run chrome with the remote debugging port enabled (e.g.9222) on windows and add the
- --remote-debugging-port=9222 setting to the config. Without it you can't use the existing session.

[DenisLanz]

Tested it again. Running Chrome with the remote debugging Port in one terminal session:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

Logging into Kleinanzeigen keeping the chrome window open. And running kleinanzeigen bot through pdm with the follwing settings in another termianl works well on macOS:

browser:
  arguments:
    - --disable-dev-shm-usage
    - --no-sandbox
    - --remote-debugging-port=9222
    - --profile-directory='Default'
  binary_location: "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
  use_private_window: false
  user_data_dir: '~/Library/Application Support/Google/Chrome'
  profile_name: 'Default'

there will be a new empty chrome window but the ads will be published in the previously opened chrome window in the background so make sure to look at that.

[TylonHH]

As I'm on a Windows maschine I run in a command line C:\Program Files\Google\Chrome\Application\chrome.exe --disable-dev-shm-usage --no-sandbox --profile-directory="Default" --remote-debugging-port=9222
I hoped that is the command wich the bot runs also. There was a strange Chrome. The default profile was not loaded. There was a new profile. Anyways, I logged in to kleinanzeigen and kept chrome open. Then I run kleinanzeigen-bot download
I new chrome opened and tried to login, captcha prompted and said I got blocked. The other windows in the background did nothing, and there was no download.

with the full command above I see this

It says that the no-sandbox is not supported

[TylonHH]

Without Sandbox and already open chrome windows

2024-01-16 00:23:11,055 [INFO] App version: 2024+8389123
2024-01-16 00:23:11,055 [INFO] Python version: 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
2024-01-16 00:23:11,064 [INFO] Loading config from [C:\TEMP\kleinanzeigen-bot\config.yaml]...
2024-01-16 00:23:11,103 [INFO]  -> found 145 categories
2024-01-16 00:23:11,103 [INFO] Creating WebDriver session...
2024-01-16 00:23:11,138 [INFO] Using Browser: GOOGLE-CHROME 120.0.6099.217 [C:\Program Files\Google\Chrome\Application\chrome.exe]
2024-01-16 00:23:12,786 [INFO]  -> Browser Profile Name: Default
2024-01-16 00:23:12,786 [INFO]  -> Custom chrome argument: --disable-dev-shm-usage
2024-01-16 00:23:12,788 [INFO]  -> Custom chrome argument: --profile-directory='Default'
2024-01-16 00:23:12,788 [INFO]  -> Custom chrome argument: --remote-debugging-port=9222
2024-01-16 00:23:12,788 [INFO]  -> Chrome binary location: C:\Program Files\Google\Chrome\Application\chrome.exe
2024-01-16 00:23:16,071 [ERROR] WebDriverException: Message: disconnected: unable to connect to renderer
Stacktrace:
	GetHandleVerifier [0x00BF6EE3+174339]
	(No symbol) [0x00B20A51]
	(No symbol) [0x00836FF6]
	(No symbol) [0x0085E500]
	(No symbol) [0x0085B9D0]
	(No symbol) [0x00857A0F]
	(No symbol) [0x0088E259]
	(No symbol) [0x0088DBBA]
	(No symbol) [0x00886DA6]
	(No symbol) [0x00861034]
	(No symbol) [0x00861F8D]
	GetHandleVerifier [0x00C94B1C+820540]
	sqlite3_dbdata_init [0x00D553EE+653550]
	sqlite3_dbdata_init [0x00D54E09+652041]
	sqlite3_dbdata_init [0x00D497CC+605388]
	sqlite3_dbdata_init [0x00D55D9B+656027]
	(No symbol) [0x00B2FE6C]
	(No symbol) [0x00B283B8]
	(No symbol) [0x00B284DD]
	(No symbol) [0x00B15818]
	BaseThreadInitThunk [0x7655FCC9+25]
	RtlGetAppContainerNamedObjectPath [0x77447C6E+286]
	RtlGetAppContainerNamedObjectPath [0x77447C3E+238]

Without the opened chrome in background, the bot start chrome with a new profile. If I click there to select my profile, there is no other profile. ?!?!?