GithubHelp home page GithubHelp logo

rusq / slackdump Goto Github PK

View Code? Open in Web Editor NEW
743.0 5.0 54.0 5.1 MB

Save or export your private and public Slack messages, threads, files, and users locally without admin privileges.

License: GNU General Public License v3.0

Go 97.41% Makefile 0.33% Dockerfile 0.24% Shell 1.86% Python 0.16%
slack archive dumper dump channels mattermost slackdump migration-tool backup slack-exports

slackdump's Introduction

Slack Dumper

Purpose: archive your private and public Slack messages, users, channels, files and emojis. Generate Slack Export without admin privileges.

Quick links:

Description

Typical use scenarios:

  • archive your private conversations from Slack when the administrator does not allow you to install applications OR you don't want to use potentially privacy-violating third-party tools,
  • archive channels from Slack when you're on a free "no archive" subscription, so you don't lose valuable knowledge in those channels,
  • create a Slack Export archive without admin access, or
  • save your favourite emojis.

There are four modes of operation (more on this in User Guide) :

  1. List users/channels
  2. Dumping messages and threads
  3. Creating a Slack Export in Mattermost or Standard modes.
  4. Emoji download mode.

Slackdump accepts two types of input (see Dumping Conversations section):

  1. the URL/link of the channel or thread, OR
  2. the ID of the channel.

Quick Start

  1. Download the latest release for your operating system from the releases page. (If you're using macOS, download Darwin executable).
  2. Unpack the archive to any directory.
  3. Run the ./slackdump or slackdump.exe executable (see note below).
  4. You know the drill: use arrow keys to select the menu item, and Enter (or Return) to confirm.

By default, Slackdump uses the EZ-Login 3000 automatic login, and interactive mode.

Note

On Windows and macOS you may be presented with "Unknown developer" window, this is fine. Reason for this is that the executable hasn't been signed by the developer certificate.

To work around this:

  • on Windows: click "more information", and press "Run Anyway" button.
  • on macOS: open the folder in Finder, hold Option and double click the executable, choose Run.

Slackord2: Migrating to Discord

If you're migrating to Discord, the recommended way is to use Slackord2 - a great tool with a nice GUI, that is compatible with the export files generated by Slackdump.

User Guide

For more advanced features and instructions, please see the User Guide.

Previewing Results

Once the data is dumped, you can use one of the following tools to preview the results:

  • SlackLogViewer - a fast and powerful Slack Export viewer written in C++.
  • Slackdump2Html - a great Python application that converts Slack Dump to a static browsable HTML, works on Dump mode files.
  • slack export viewer - Slack Export Viewer is a well known viewer for slack export files.

Using as a library

Download:

go get github.com/rusq/slackdump/v2

Example

package main

import (
  "context"
  "log"

  "github.com/rusq/slackdump/v2"
  "github.com/rusq/slackdump/v2/auth"
)

func main() {
  provider, err := auth.NewValueAuth("xoxc-...", "xoxd-...")
  if err != nil {
      log.Print(err)
      return
  }
  sd, err := slackdump.New(context.Background(), provider)
  if err != nil {
      log.Print(err)
      return
  }
  _ = sd
}

See Go Reference

Using Custom Logger

Slackdump uses a simple rusq/dlog as a default logger (it is a wrapper around the standard logger that adds Debug* functions).

If you want to use the same default logger that Slackdump uses in your application, it is available as logger.Default.

No doubts that everyone has their own favourite logger that is better than other miserable loggers. Please read below for instructions on plugging your favourite logger.

Logrus

Good news is logrus can be plugged in straight away, as it implements the logger.Interface out of the box.

lg := logrus.New()
sd, err := slackdump.New(context.Background(), provider, WithLogger(lg))
  if err != nil {
      log.Print(err)
      return
  }
}

Glog and others

If you need to use some other logger, such as glog, it is a matter of wrapping the calls to satisfy the logger.Interface (defined in the logger package), and then setting the Logger variable in slackdump.Options (see options.go), or using WithLogger option.

FAQ

Q

Do I need to create a Slack application?

A

No, you don't. Just run the application and EZ-Login 3000 will take care of the authentication or, alternatively, grab that token and cookie from the browser Slack session. See User Guide.

Q

I'm getting "invalid_auth" error

A

Go get the new Cookie from the browser and Token as well.

Q

Slackdump takes a very long time to cache users

A

Disable the user cache with -no-user-cache flag.

Q

How to read the export file?

A

For Slack Workspace Export, use SlackLogViewer which is extremely fast with an advanced search function, or slack export viewer which is a Python application and runs in a browser. For the generic dump files, see examples directory for some python and shell examples.

Q

My Slack Workspace is on the Free plan. Can I get data older than 90-days?

A

No, unfortunately you can't. Slack doesn't allow to export data older than 90 days for free workspaces, the API does not return any data before 90 days for workspaces on the Free plan.

Thank you

Big thanks to all contributors, who submitted a pull request, reported a bug, suggested a feature, helped to reproduce, or spent time chatting with me on the Telegram or Slack to help to understand the issue and tested the proposed solution.

Also, I'd like to thank all those who made a donation to support the project:

  • Vivek R.
  • Fabian I.
  • Ori P.
  • Shir B. L.
  • Emin G.
  • Robert Z.
  • Sudhanshu J.

Bulletin Board

Messages that were conveyed with the donations:

  • 25/01/2022: Stay away from TheSignChef.com, ya hear, they don't pay what they owe to their employees.

slackdump's People

Contributors

dependabot[bot] avatar eau-u4f avatar enterjazz avatar errge avatar fitzyjoe avatar jhult avatar kolsys avatar omarcostahamido avatar robws avatar rusq avatar yarikoptic avatar yuvipanda avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

slackdump's Issues

Update attachment links to point to downloaded files.

slackdump is incredibly helpful to me. Many thanks for building this utility!

I have a question about parsing the .json output and the example python code helped me greatly. Currently, a http link is recorded for attached files. Is there a way of automatically downloading attached files (such as images, videos and text files) so that they also display nicely in the python output?

I also wonder if there is an option, since attachments are already downloaded into a folder, to have the links point to those files instead?

Originally posted by @jingxlim in #70

Exceeding rate limit on channels with more than 100 messages

Describe the bug
Script dumps nothing when slack rate limit is exceeded.

To Reproduce
Tried it with and without the -limiter-boost:

slackdump-darwin % ./slackdump -r text CxxxxxM8
2022/02/16 14:04:41 > caching users...
2022/02/16 14:04:41 > caching channels, might take a while...
2022/02/16 14:04:49 dumping channel: "CxxxxxxM8"
2022/02/16 14:05:04 request #1, fetched: 100, total: 100
2022/02/16 14:05:13 channel "CxxxxxxxM8": slack rate limit exceeded, retry after 1s
2022/02/16 14:05:13 job finished, dumped 0 channels
slackdump-darwin % ./slackdump -r text CxxxM8 -limiter-boost=0
2022/02/16 14:09:17 > caching users...
2022/02/16 14:09:17 > caching channels, might take a while...
2022/02/16 14:09:24 dumping channel: "CxxxM8"
2022/02/16 14:09:40 request #1, fetched: 100, total: 100
2022/02/16 14:09:57 request #2, fetched: 100, total: 200
2022/02/16 14:10:14 request #3, fetched: 100, total: 300
2022/02/16 14:10:24 channel "CxxxM8": slack rate limit exceeded, retry after 1s
2022/02/16 14:10:24 dumping channel: "-limiter-boost=0"
2022/02/16 14:10:25 channel "-limiter-boost=0": channel_not_found
2022/02/16 14:10:25 job finished, dumped 0 channels

Expected behavior
App finishes the job or at least gives me the currently fetched messages.

slackdump -c does not work

Hello.
slackdump -c does not work for me. My secrets.txt file has the two parameters SLACK_TOKEN and COOKIE provided but I am getting invalid_auth.

Here is what I get:

❯ ./slackdump -c
2022/02/04 18:43:32 > caching users...
2022/02/04 18:43:32 caching users for the first time
2022/02/04 18:43:33 error fetching users: invalid_auth

how to export a specific thread?

I came across this great tool, but I am a bit puzzled as to how to export/dump a specific thread.
Channels can be extremly large and I am intersted in a specific thread. How can I export that thread alone?

In an org with 2000 channels slackdump -c would not give me a specific thread id, or at least I probably won't find it in there.
When I copy a link to a thread in Slack, I get 2 ids in the link. Can I use that info to dump a specific thread? If yes, how?

JSON marshalling error during export

Describe the bug
Using v2.0.0 I get the following error in the middle of the export of my workspace:
channels: error: failed dumping "" (DHRCM1N81): json: cannot unmarshal number into Go struct field Message.messages.blocks of type string

The channel ID is the one from the outlook_calendar. I still get the error after disconnecting from Slack. How can I exclude this channel to finish the export successfully?

Desktop (please complete the following information):

  • OS: Windows 10

When exporting conversations, only export participating users

Is your feature request related to a problem? Please describe.

When a specific conversation is exported via the -c argument, the users.json file will still contain the entire members directory. In popular Slacks this can easily be megabytes of data even when only a conversation between two members gets exported. This data may also contain PII that I'm not allowed to store outside of Slack.

Describe the solution you'd like

When exporting conversations, allow the export of users to be limited to participating members.

Describe alternatives you've considered

  • I tried to export with the -no-user-cacheargument, but it will fail with application error: failed to create an index: empty users data base.
  • I could manually read and purge the users.json, but this feels error prone.

Additional context

  • I've deliberately left out mentions, because for archival purposes their user data shouldn't be relevant (I think?).
  • Restricting to participants may also speed up the export, as fetching the entire user directory can take a minute. I realize though the current architecture may not support this boost.

Fetching only starred and/or reacted items

Is your feature request related to a problem? Please describe.
Some people organise their information using stars and reactions. Currently there's no way to get only the starred and/or reacted items, and this might be a problem on a large workspaces, when the only choice would be to manually assemble the list of items to download, or download the large workspace in full.

Describe the solution you'd like
Implement the ability to export only starred and/or reacted items (messages, replies, attachments)

Describe alternatives you've considered
Getting with curl and assembling list manually.

Additional context
Fork from #109

Export max size limit? "Bad magic number for file header"

Describe the bug
I am exporting a bunch of channels, DMs etc. The export finishes without errors.
When I try to open the zip file created, I get this error.

Windows cannot open the folder.
The Compressed (zipped) Folder 'C:\Data\SlackDump\my-workspace.zip' is invalid.

Other software gives the error in the title.

One thing I have noticed is that the zip is roughly 1 GB in size in size, hence the question on size limits.

I have not had the issue with exporting a couple of channels and DMs where the size was much smaller.

To Reproduce
Steps to reproduce the behavior:

  1. .\slackdump.exe -export my-workspace.zip -download -user-cache-age 96h0m0s -user-cache-file u.cache G02*** D02*** C02*** ...

Expected behavior
Usable export file.
Using the suggested slack-export-viewer works with smaller size archive.

Output
The output reports completion but curiously logs an additional save afterwards.

2022/07/18 14:11:27   thread fetch complete, total: 6
2022/07/18 14:11:27 messages request #    3, fetched:   31 (threads: 14, files: 8), total:      431 (speed:   2.22/sec, avg:   2.54/sec)
2022/07/18 14:11:27 messages fetch complete, total: 431
2022/07/18 14:11:33 completed, time taken: 26m30.7357232s
2022/07/18 14:11:33 file "F02***-image.png" saved to abcd\attachments: 24999 bytes written

Desktop (please complete the following information):

  • OS: Windows 11
    22H2 (OS Build 22622.290)

Running

PS C:\Data\SlackDump> .\slackdump.exe -V
Slackdump v2.0.1 Copyright (c) 2018-2022 rusq (build: 1c46ed8)

v2.0.1

Ran a smaller export with the trace option.

2022/07/18 14:42:57   thread fetch complete, total: 2
2022/07/18 14:42:58   thread request #    1, fetched:    4, total:        4, process results:  (speed:  10.67/sec, avg:  10.67/sec)
2022/07/18 14:42:58   thread fetch complete, total: 4
2022/07/18 14:42:58 messages request #    1, fetched:   43 (threads: 10, files: 14), total:       43 (speed:  11.80/sec, avg:  11.80/sec)
2022/07/18 14:42:58 messages fetch complete, total: 43
2022/07/18 14:43:04 completed, time taken: 24.2782739s
2022/07/18 14:43:04 error saving "F0***K-Access for abcd.docx" to cba\attachments: write my-workspace2.zip: file already closed
2022/07/18 14:43:04 error saving "F0***W-image.png" to cba\attachments: write my-workspace2.zip: file already closed
2022/07/18 14:43:04 error saving "F0***7-image.png" to cba\attachments: write my-workspace2.zip: file already closed
2022/07/18 14:43:04 error saving "F0***Y-image.png" to cba\attachments: context canceled
2022/07/18 14:43:04 error saving "F0***A-image.png" to cba\attachments: write my-workspace2.zip: file already closed
2022/07/18 14:43:04 error saving "F0***9-Untitled.js" to cba\attachments: context canceled
PS C:\Data\SlackDump> dir

Support auth for multiple workspaces

Is your feature request related to a problem? Please describe.

I would like to use this with multiple different Slack URLs without having keep logging in to each different one.

Describe the solution you'd like
When saving the auth associate it with the URL in the file in which you save it. Then when running slackdump with no args and it gets to the Slack Workspace Name prompt, it would list the ones authorized but also allow selecting "Authenticate a new Workspace."

This would probably make the -auth-reset option mostly moot.

Describe alternatives you've considered
I can't think of one other than manually logging in and out.

Additional context
I'd be happy to consider writing this feature — I am professionally working as a Go developer — but I'd want to know that you would bless the idea and offer your tips in advance for how you would want it done so that I don't waste time producing a PR you would not accept.

Internal Server Error when exporting a very large thread (800+ messages)

Describe the bug
I'm trying to export a thread with more than 800 messages (as of today) but I get an error when I run the following command:
./slackdump -r text https://xyz.slack.com/archives/RK9N4AFRB/p165158407

This thread mostly contains images, some text messages, some links, standard/custom emojis/reactions.

I can export smaller threads (fewer than 5 messages) that are either older or newer from the same private channel without any problem.

I'm using version 1.3.5.

Output

2022/05/05 13:31:54 > checking user cache...
2022/05/05 13:32:07 error processing: "https://xyz.slack.com/archives/RK9N4AFRB/p165158407" (conversation will be skipped): slack server error: 500 Internal Server Error
2022/05/05 13:32:07 job finished, dumped 0 item(s)

Desktop (please complete the following information):

  • OS: Windows 10

AttributeError when opening export with slack-export-viewer

I just setup slackdump and slack-export-viewer today. The export was successful but when I try to open it with the viewer, it errored out with an AttributeError. Any ideas about this?

20220728.zip extracting to /var/folders/9q/nm0mln8j3m38fk8621xn7l580000gn/T/_slackviewer/4ccc26e8ef2c6b952e48cbe2c3639cc1aa885400...
20220728.zip extracted to /var/folders/9q/nm0mln8j3m38fk8621xn7l580000gn/T/_slackviewer/4ccc26e8ef2c6b952e48cbe2c3639cc1aa885400
Traceback (most recent call last):
File "/Users/sjuneja/.pyenv/versions/3.9.4/bin/slack-export-viewer", line 33, in
sys.exit(load_entry_point('slack-export-viewer==1.1.4', 'console_scripts', 'slack-export-viewer')())
File "/Users/sjuneja/.pyenv/versions/3.9.4/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/Users/sjuneja/.pyenv/versions/3.9.4/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/sjuneja/.pyenv/versions/3.9.4/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sjuneja/.pyenv/versions/3.9.4/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/sjuneja/.pyenv/versions/3.9.4/lib/python3.9/site-packages/slackviewer/main.py", line 61, in main
configure_app(app, archive, channels, no_sidebar, no_external_references, debug)
File "/Users/sjuneja/.pyenv/versions/3.9.4/lib/python3.9/site-packages/slackviewer/main.py", line 24, in configure_app
top.channels = reader.compile_channels(channels)
AttributeError: 'LocalStack' object has no attribute 'channels'

attachments/images preview/thumbnails download

Is your feature request related to a problem? Please describe.
Thumbnails for images in channels etc are not downloaded and instead refer to the original thumbnail on slack.
Due to this when viewing the messages in the group, the thumbnails are broken and do not display due to CORB issues and mimetype issues.
Additionally, expecting an archival tool to pull down all such assets or at least have an option for the same.
Pretty soon my company is moving away from slack and so we won't have access to the original images etc.

Describe the solution you'd like
Download thumbnails as well just like the actual images/attachments.

channels.json empty/null

Describe the bug
When running slackdump, channels.json has null as the only content.
This results in the export not being usable with a viewer.

To Reproduce
Steps to reproduce the behavior:

  1. Run slackdump like this './slackdump -v -export bug -download' with 1 or more channels ids.

Expected behavior
Correct channels.json generation.

Desktop (please complete the following information):

  • OS: Windows 11 insider/wsl2/Ubuntu 22.04/slackdump 2.1.1

Windows version 2.0.3 also resulted in the same issue.
Just tried with 2.1.2 on windows and it also resulted in the same issue.

Dumping some obscure thread within the channel returns invalid JSON: invalid character '<' looking for beginning of value

Describe the bug
An error occurs while trying to dump a DM. Dump runs fine until it meets with the obscure thread. Looking at the error, it seems that HTML is returned by the API.

User is using browser auth, so all set of cookies is set.

To Reproduce
Steps to reproduce the behavior:

  1. Run slackdump:
./slackdump DXXXXXXX

Expected behavior
Channel is saved to JSON.

Output

failed to dump channel:thread DXXXXXXX:1666555444.333222: invalid character '<' looking for beginning of value after 1 attempts

Desktop (please complete the following information):

  • OS: Windows

Additional context
Add any other context about the problem here.

Export to Markdown

Describe the solution you'd like

slackdump -r markdown

with images

Describe alternatives you've considered
I can find a tool which convert text to markdown.

Additional context
I believe there will be go libraries to convert Slack format (HTML in json?) to Markdown

slack rate limit exceeded (again)

I'm seeing a similar problem to #1 when I try to download a single channel (it manages to get 820 files, then stops with the message "slack rate limit exceeded, retry after 1s"). Would it be possible to provide a command-line argument for adjusting the rate of requests? I got this error on linux, then tried downloading the latest Windows build in the hopes that the extra delay imposed by my VPN connection might help (it didn't).

Originally posted by @shi-bmz in #1 (comment)

exceeding amount of rate limit retry attempts on large amount of channels (known issue)

if you're experiencing this, the current workaround on version 1.3.0 and prior is:

./slackdump -c -limiter-boost=0

This make us stay within the Tier 2.

Describe the bug
./slackdump -c on large amount of channels gets rate limited and we exceed the number of retry attempts (default: 3)

Expected behavior
We're not getting rate limited, or if we are, we're brave enough to withstand any amount of retries (or let the user control those)

Desktop (please complete the following information):

  • OS: all
  • Version: v1.3.0 (and below)

Implement builds/releasing in Github Actions/Travis CI to ensure that release binaries builds can be validated

Just, FYI, something in the windows binary releases is getting flagged by some of the (more exotic) scanners in VirusTotal (Cylance, Cynet, MaxSecure, and SecureAge APEX). Reads clean on everything else. Likely a false positive, but, figured it was worth mentioning. (Maybe it's a compiler setting or an AV-AI confused by Go?)

To Reproduce
Upload binary to VirusTotal, see red text, panic.

Expected behavior
That...not happening.

Panics on 2GB+ ZIP archives

Describe the bug
v2.0.2 Panics on huge archives in bufio.go

bufio.(*Writer).Flush(0xc00007e000)
	/usr/local/Cellar/go/1.18.3/libexec/src/bufio/bufio.go:634 +0x171

Archive size:

-rw-r--r--  1 rustam  staff  2376101723 21 Jul 21:51 test.zip

To Reproduce
Steps to reproduce the behavior:

  1. go run ./cmd/slackdump -f -export 'test.zip' https://x.slack.com/archives/x

Expected behavior
A clear and concise description of what you expected to happen.

Output

2022/07/21 21:51:14 file "x.png" saved to random/attachments: 555722 bytes written
panic: runtime error: slice bounds out of range [:4201] with capacity 4096

goroutine 56 [running]:
bufio.(*Writer).Flush(0xc00007e000)
	/usr/local/Cellar/go/1.18.3/libexec/src/bufio/bufio.go:634 +0x171
bufio.(*Writer).Write(0xc00007e000, {0xc00a940200?, 0x0?, 0xc0024578e0?})
	/usr/local/Cellar/go/1.18.3/libexec/src/bufio/bufio.go:672 +0xd8
archive/zip.(*countWriter).Write(0xc0001282a0, {0xc00a940200?, 0x7?, 0x2?})
	/usr/local/Cellar/go/1.18.3/libexec/src/archive/zip/writer.go:601 +0x2e
archive/zip.(*countWriter).Write(0xc00026e108, {0xc00a940200?, 0xc002620420?, 0xc002620450?})
	/usr/local/Cellar/go/1.18.3/libexec/src/archive/zip/writer.go:601 +0x2e
compress/flate.(*huffmanBitWriter).write(...)
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/huffman_bit_writer.go:136
compress/flate.(*huffmanBitWriter).writeCode(0xc00a9401e0?, {0xe180?, 0x5ca?})
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/huffman_bit_writer.go:347 +0xe5
compress/flate.(*huffmanBitWriter).writeTokens(0xc00a9401e0, {0xc00e828000, 0x4001, 0x2d541400?}, {0xc00a930d80, 0x11e, 0x100ea1d?}, {0xc005cae280, 0x1e, 0x1e})
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/huffman_bit_writer.go:583 +0xb9
compress/flate.(*huffmanBitWriter).writeBlock(0xc00a9401e0, {0xc00e828000?, 0x20?, 0xffffffffffffc090?}, 0x0, {0xc00cdf0000, 0x4090, 0x10000})
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/huffman_bit_writer.go:495 +0x490
compress/flate.(*compressor).writeBlock(0xc00e786000, {0xc00e828000?, 0x10fced5?, 0x30?}, 0x0?)
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/deflate.go:170 +0x9c
compress/flate.(*compressor).deflate(0xc00e786000)
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/deflate.go:509 +0x59b
compress/flate.(*compressor).write(0xc00e786000, {0xc00e6f0000?, 0x14ed6, 0xca414fff?})
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/deflate.go:554 +0x82
compress/flate.(*Writer).Write(...)
	/usr/local/Cellar/go/1.18.3/libexec/src/compress/flate/deflate.go:712
archive/zip.(*pooledFlateWriter).Write(0xc0007437d0?, {0xc00e6f0000?, 0x14ed6?, 0xc002457c28?})
	/usr/local/Cellar/go/1.18.3/libexec/src/archive/zip/register.go:51 +0xc5
archive/zip.(*countWriter).Write(...)
	/usr/local/Cellar/go/1.18.3/libexec/src/archive/zip/writer.go:601
archive/zip.(*fileWriter).Write(0xc000020370, {0xc00e6f0000, 0x14ed6, 0x1fe00})
	/usr/local/Cellar/go/1.18.3/libexec/src/archive/zip/writer.go:533 +0x97
bytes.(*Reader).WriteTo(0xc000358810, {0x15b9800?, 0xc000020370?})
	/usr/local/Cellar/go/1.18.3/libexec/src/bytes/reader.go:144 +0x87
io.copyBuffer({0x15b9800, 0xc000020370}, {0x15b98a0, 0xc000358810}, {0x0, 0x0, 0x0})
	/usr/local/Cellar/go/1.18.3/libexec/src/io/io.go:408 +0x16e
io.Copy(...)
	/usr/local/Cellar/go/1.18.3/libexec/src/io/io.go:385
github.com/rusq/slackdump/v2/fsadapter.(*ZIP).WriteFile(0x15bcfc8?, {0xc006bae000?, 0x27?}, {0xc00e6f0000, 0x14ed6, 0x1fe00}, 0x2457dc8?)
	/Users/rusq/gdrive/dev/go/github.com/rusq/slackdump/fsadapter/zipfs.go:64 +0x178
github.com/rusq/slackdump/v2/downloader.(*Client).saveFile(0xc000120070, {0x15bcfc8, 0xc00011c090}, {0xc0009d0810, 0x12}, 0xc005055200)
	/Users/rusq/gdrive/dev/go/github.com/rusq/slackdump/downloader/downloader.go:238 +0x270
github.com/rusq/slackdump/v2/downloader.(*Client).worker(0xc000120070, {0x15bcfc8, 0xc00011c090}, 0xc000176000)
	/Users/rusq/gdrive/dev/go/github.com/rusq/slackdump/downloader/downloader.go:176 +0x309
github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers.func1(0x0?)
	/Users/rusq/gdrive/dev/go/github.com/rusq/slackdump/downloader/downloader.go:155 +0x65
created by github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers
	/Users/rusq/gdrive/dev/go/github.com/rusq/slackdump/downloader/downloader.go:154 +0x65
exit status 2

Desktop (please complete the following information):

  • OS: macOS

Additional context
Add any other context about the problem here.

`go install` gets error

Describe the bug
I am trying to make use of this package as a library, and installing produces an error

../../../../pkg/mod/github.com/rusq/slackdump/[email protected]/slackdump.go:95:51: undefined: slack.OptionCookieRAW

To Reproduce

  1. go install / go build

Expected behavior
Successful install

Desktop (please complete the following information):

  • OS: macOS monterey

Release for Mac OS X?

I downloaded the Linux release to attempt to execute on my Mac OS X machine. I've checked the file settings, and it's executable, but when I try to run it, I keep getting "slackdump: command not found".

Have I missed a step during unpacking? (I simply unzipped the file and proceeded). Or did I download the wrong release file?

I'm new to Go, so I apologize if I'm missing something super obvious. I'm very excited about this package's functionality!

Simplify manual token extraction steps

Describe the bug

It seems that the instructions for manual token extraction are too complicated. As far as I'm aware dealing with network requests isn't necessary, as the required information can be found in localStorage.

This snippet, when executed in the dev tools console, extracts the current team ID from the URL and retrieves the token:

JSON.parse(localStorage.localConfig_v2).teams[document.location.pathname.match(/^\/client\/(T[A-Z0-9]+)/)[1]].token

Or a bit more readable:

JSON.parse(
  localStorage.localConfig_v2
).teams[
  // Conservative matching of the team ID
  document.location.pathname.match(/^\/client\/(T[A-Z0-9]+)/)[1]
].token

User list does not include custom fields

Describe the bug

The command slackdump -v -list-users -r json does not include the custom fields in user profiles.

For example, a request of the form
https://SUBDOMAIN.slack.com/api/search.modules?_x_id=1111111-11111111.551&_x_csid=2222-33333&slack_route=12345678&_x_version_ts=ABCDEFGH&_x_gantry=true&fp=7d

returns profile data in the form

{
    "iid": "33332999-7520-4444-9a6a-43fee1d42caa",
    "id": "UDJMAGZZZ",
    "profile": {
        "title": null,
        "phone": null,
        "skype": "",
        "real_name": "Jon Doe",
        "real_name_normalized": "Jon Doe",
        "display_name": "jon.doe",
        "display_name_normalized": "jon.doe",
        "fields": {
            "Zz11HKQG22": {
                "value": "role",
                "alt": ""
            },
            "Zz11KKCF22": {
                "value": "2018-10-22",
                "alt": ""
            },
            "Zz11JK4P22": {
                "value": "location",
                "alt": ""
            },
            "Zz11JKQ722": {
                "value": "https:\/\/example.com\/people\/343000",
                "alt": "text"
            },
            "Zz113KK222": {
                "value": "U4GGSS333",
                "alt": ""
            },
            "Zz11BKG822": {
                "value": "function",
                "alt": ""
            },
            "Zz118KA022": {
                "value": "project name",
                "alt": ""
            },
            "Zz115KTT22CB": {
                "value": "747578",
                "alt": ""
            }
        },
        "status_text": "",
        "status_emoji": "",
        "status_emoji_display_info": [],
        "status_expiration": 0,
        "avatar_hash": "cccccccccc",
        "image_original": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_original.jpg",
        "is_custom_image": true,
        "email": "[email protected]",
        "first_name": "Jon",
        "last_name": "Doe",
        "image_24": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_24.jpg",
        "image_32": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_32.jpg",
        "image_48": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_48.jpg",
        "image_72": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_72.jpg",
        "image_192": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_192.jpg",
        "image_512": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_512.jpg",
        "image_1024": "https:\/\/avatars.slack-edge.com\/2022-02-12\/abcdefgh-7b6_1024.jpg",
        "status_text_canonical": "",
        "team": "TTTTTTT"
    },
    "username": "jon.doe",
    "is_restricted": null,
    "is_ultra_restricted": null,
    "phone": "",
    "top_terms": false,
    "similar": false
}

The key fields is not present in the output of slackdump -v -list-users -r json.

Is there a way to have those fields included? What would need to be changed in the code in order to get that data?

Batch export

There was a suggestion in telegram channel to add an option to export all the data from a slack workspace (all channels, all users).

For Slack users with blank display names, include the attribute "display_name": ""

Describe the bug
[This is arguably not a bug, but if using the exports as a means to import the data into another tool that expects the Slack format, then it is.]
If a Slack user's Display Name isn't set, then slackdump does not include an attribute for it in the exports.

To Reproduce
slackdump -export <dir> <Channel-ID> where 1 or more users in the conversation have blank display names.

Expected behavior
Include a "display_name": "", in the JSON. The native Slack exporter does this.

Example JSON Output
The user_profile block in the JSON should look like (this is a sample from the Slack exporter):

"user_profile": {
            "avatar_hash": "g38ae5c8db8x",
            "image_72": "https:\/\/secure.gravatar.com\/avatar\/f38ae5c8db89d6e4a997644ad609b7c0.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0008.png",
            "first_name": "KJH",
            "real_name": "KJH",
            "display_name": "",
            "team": "TGFDWL5H0",
            "name": "kjh",
            "is_restricted": false,
            "is_ultra_restricted": false
        },

I don't believe the last 2 attributes are strictly necessary, because I was able to perform a successful import (using a Slack to Discord tool) by simply adding in the "display_name": "", attribute.

Error "failed to dump channel C0XXXXXXX: json: cannot unmarshal number into Go struct field ..."

Slackdump v2.2.0-alpha hangs in the middle.

  1. Run slackdump -export archive.zip -export-type standard -download
  2. Several channel posts and files are successfully stored in a ZIP file (~500 MB)
  3. The program hangs with the following error
...
2022/08/29 12:13:02 file "document.docx" saved to channel_name\attachments: 19000 bytes written
2022/08/29 12:13:02 application error: export error: channels: error: failed to dump "test_channel_name" (C0XXXXXXX): callback error: failed to dump channel C0XXXXXXX: json: cannot unmarshal number into Go struct field Message.messages.blocks of type string

The channel name has Kanji charactors, Katakana charactors, alphabets, numbers, and underscores.
This channel has been archived and unarchived several times on Slack.

The ZIP files cannot be viewed with a Windows default ZIP viewer, but I was able to view the ZIP file using 7zip viewer.

  • OS: Windows 10

I used slackdump for 30 other Slack workspaces due to the 90 days limits, and they all worked fine.
Thanks for the awsome software.

invalid_auth on some Slack workspaces

Describe the bug
Using enterprise/paid version of Slack, after configuring a secrets.txt file, running slackdump -u, it is unable to download the users or anything else:
2022/03/16 20:19:25 > checking user cache...
2022/03/16 20:19:25 caching users for the first time
2022/03/16 20:19:25 error fetching users: couldn't fetch users

verbose output from slackdump -v -u:
2022/03/16 20:22:22 slackdump.go:88: > checking user cache...
2022/03/16 20:22:22 users.go:37: caching users for the first time
2022/03/16 20:22:22 main.go:65: error fetching users: couldn't fetch users

To Reproduce
Steps to reproduce the behavior:
When I used chrome to get the value of the authorization cookie, I did have to do URL decode so that it would appear as a base64 value. It does not appear that it is complaining about authorization.

Expected behavior
A clear and concise description of what you expected to happen.
Expected it to list the users

Output
If applicable, add output to help explain your problem.
Output is shown above.

Desktop (please complete the following information):

  • OS: [e.g. macOS]
    Windows 11 using released binaries.

Additional context
Add any other context about the problem here.

Ignoring errors

Describe the bug
while bugs are encountered, export process terminates
To Reproduce
Steps to reproduce the behavior:

  1. Run slackdump like this '...'
    slackdump -f -export MyDump @channelList.txt
    Expected behavior
    Continue or prompt if to ignore error.
    Output
    2022/08/16 13:53:39 application error: export error: failed to dump "xyz" (xxx): callback error: failed to dump channel xxx: strconv.Atoi: parsing "null": invalid syntax

Desktop (please complete the following information):

  • OS: [e.g. macOS]
    Windows
    Additional context
    Same problem occurs if there is temporary slack down time e.g. 503 error.
    For larger exports it may take time to process.
    It would be good to expose flags to ignore error and/or retry X times after Y time then ignore or exit.
    Now i need to forgo entire channel as it is failing. Also had to write script that compares continent if other channels fail (not fatally) and can be restarted later.

Otherwise gj bro!

EZ-Login 3000: Automatically dismiss or suppress the cookie banner

Is your feature request related to a problem? Please describe.

The EZ-Login 3000 is amazing and makes the login process dead simple. A very minor downside here is that it creates a fresh browsing context that will always show the cookie consent banner. Depending on the window size the banner may initially cover part of the login form.

Describe the solution you'd like

On Slack's own login form, dismiss the banner by programmatically by either:

  • clicking the reject button programmatically via document.getElementById('onetrust-reject-all-handler').click(),
  • or by setting the OptanonAlertBoxClosed cookie with a recent date value.

Possible to export slack data with API token for migrating to mattermost?

Is your feature request related to a problem? Please describe.
Currently I'm trying to export the all slack messages including private and group message to mattermost.

Describe the solution you'd like
I think having an option when API token(Admin) is avaiable to make a export full workspace would be great with this tool.

EZ Login

Make it easier to grab the token+cookie.

Options:

  • ms-playwright
  • chromedp

Presence

The second request is to get only active users (where presence:active in results or the user IDs are in the presence_active_ids list).
It would be super awesome if the request to users/list had a “filter” for it, but I haven’t found the magic keyword
There is a “filter” keyword to the request. Initially it’s “people”. And when they use users/search, the filter gets more complex, “people AND NOT deactivated AND NOT external”

not dumping users with guest role

Describe the bug
Guest users information is not being dumped to users.json.

To Reproduce
Steps to reproduce the behavior:

  1. Join a workspace with guest/multi-channel guest users
  2. Dump it
  3. Check users.json
  4. All their messages will show with only their ID in a full dump, and users.json doesn't have a record for those users.

Expected behavior
Populated guest users in users.json

Output
Not applicable

Desktop (please complete the following information):

  • OS: [Windows 10 x64]

Additional context
I tested this in a workspace that have about 400 normal users and 2000 guests. Got data for all normal users and no data from guests. It shows fine in browser/slack desktop client.

Program won't open on Windows

Describe the bug
When I try to run slackdump.exe, a window opens and then immediately disappears.

To Reproduce
I open the file.

Expected behavior
I expect the program to open and run but there seems to be a compatibility issue that the windows compatibility wizard can't identify.

Desktop (please complete the following information):

  • OS: Windows

Additional context
Sorry, I know this is probably too vague and submitted in the wrong place, but I would love to use this tool and have no idea how to resolve this issue. Thank you.

I am using the IDs to export the chat. But the response is always the channel does not exist.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Hitting 429 on a large channel with files

Telegram channel:

hi there, thought I’d ask - in v1.1.1/latest - it looks like in large channels, when I get ratelimited by slack, slackdump seems to return the following:

2022/01/18 23:59:13 channel “xxxx”: slack rate limit exceeded, retry after 1s
2022/01/18 23:59:13 job finished, dumped 0 channels

We should either die with error or retry, and we do neither.

Follow up:

Initially tried to dump multiple channels but switched to one by one, still ran into the ratelimuting after waiting roughly an hour. It seems to vary how quickly I get rate limited, sometimes 1000 messages, sometimes 500

+ characters in d cookie value

Discussed in #65

Originally posted by pican79 May 26, 2022
Hi,
I've just discovered that if the d cookie value contains 1 (or more) + character, I always get the invalid_auth error when I use the manual authentication.
NB: my Slack workspace uses SSO.
To avoid that problem, I have to urlencode the character, ie replace + by %2B.

If I'm right, do you think it's something that could be documented here?
https://github.com/rusq/slackdump/blob/master/doc/login-manual.rst

I almost went crazy when slackdump (v2.0.0-beta.2) stopped working all of a sudden.
Thanks.

Unable to run the script

Describe the bug
go get command is deprecated since v1.17, go install command is now being recommended instead.
go install doesn't support replace directives that are being used in go.mod file.

To Reproduce
Steps to reproduce the behavior:

  1. Run go install github.com/rusq/slackdump/v2@latest

Output
go: github.com/rusq/slackdump/v2@latest (in github.com/rusq/slackdump/[email protected]): The go.mod file for the module providing named packages contains one or more replace directives. It must not contain directives that would cause it to be interpreted differently than if it were the main module.

Desktop:

  • OS: macOS version 12.3.1
  • go version: go1.18.2 darwin/amd64

Slack-export compatibility

v2.0.0-rc

Also, it is expected that attachment won't be imported. When one creates an export using the Slack workspace export, what happens is 1) Slack generates a special "export token" 2) all export json conversation files links are updated to include this token in the URL. So the files are still hosted on slack and not physically downloaded into the ZIP archive generated by slack. I suspect, that when one imports the conversations, Slack tries to access those links to remap the files to the new workspace. Potentially the solution would be to start a web server that Slack can see, update all file links in conversations to this sever address to let the new Slack workspace download it, if that was happens, but this is just my guess

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.