nicm / fdm Goto Github PK

fdm source code

C 85.78% Shell 0.06% Awk 0.26% Yacc 7.72% Makefile 0.37% M4 0.54% Roff 5.27%

fdm's Introduction

fdm is a program designed to fetch mail from POP3 or IMAP servers, or receive
local mail from stdin, and deliver it in various ways.

As of fdm 0.9, the caching has been changed, so Berkeley DB is no longer
required.

As of fdm 1.4, TDB is a required dependency.

See the included MANUAL file and the fdm(1) and fdm.conf(5) man pages for
installation and usage instructions. Some example configurations are included
in the examples directory.

Feedback, bug reports, suggestions, etc, are welcome.

Nicholas Marriott <[email protected]>

fdm's People

Contributors

Stargazers

Watchers

fdm's Issues

socket error after successful POP3 session

While moving from fetchmail+procmail to fdm, I found the following when using fdm -vvvv kind of strange. So far, this issue appears cosmetic, but I wanted to ask anyway.

On a successful session (but no new mail), I see:

parent: started, pid is 44521
parent: 0 children, 0 dead children
parent: child 44522 (mambo) started
mambo: fetch started, pid 44522
...
< +OK 0 0
mambo: fetch loop start
mambo: calling fetch state (0x420d60, flags 0x02)
mambo: fetch, block
mambo: queued 0; blocked 0; flags 0x02
mambo: polling: 2, timeout=900000
> QUIT
mambo: fetch loop start
mambo: calling fetch state (0x421010, flags 0x02)
mambo: fetch, block
mambo: queued 0; blocked 0; flags 0x02
mambo: polling: 2, timeout=900000
< +OK Logging out.
mambo: fetch loop start
mambo: calling fetch state (0x421010, flags 0x02)
mambo: fetch, exit
mambo: 0 messages processed in 0.723 seconds
mambo: finished processing. exiting
mambo: sending exit message to parent
mambo: waiting for exit message from parent
parent: got message type 1, id 0 from child 44522
parent: sending exit message to child 44522
parent: 1 children, 0 dead children
parent: 1 children, 0 dead children
parent: child 44522 socket error
parent: 1 children, 0 dead children
parent: waiting for children
parent: child 44522 returned 0
parent: waiting for children
parent: waiting for children
parent: finished, total time 0.724 seconds

Question: why is socket error shown? At this phase, the socket closure and underlying FIN/FIN+ACK handshake should be successful, so I don't know why fdm insists this is a "socket error".

Client OS: FreeBSD 11.4-STABLE running fdm version 2.0 from pkg/ports
Remote OS: FreeBSD 11.3-STABLE running Dovecot 2.x -- however, this same problem happens against non-FreeBSD and non-Dovecot servers (such as Comcast's IMAP/POP3 server), so I do not believe the server OS or POP3 daemon software is relevant.

missing pcre detection in autoconf

fdm has support for PCRE, but the configure script neither detects nor allows to enable it. (In particular, you need CFLAGS=-DPCRE and LIBS=-lpcre)

deleting messages from server

sorry if i missed it, but i couldn't find any information about
this in the documentation. i would like to keep messages on the
server for a while, say 14 days, and then delete them. this
will ensure that i will have made a backup of my local files in
the meantime.

there was an option for this in getmail, but here there doesn't
seem to be any way of doing it. it seems that, unless the -k
flag is passed to fdm, messages are immediately deleted from the
server after having been (matched and) fetched. or?

proxy setting for specific mail server.

fdm manual says,

 proxy url
             This instructs fdm(1) to proxy all connections through url.  HTTP and SOCKS5 proxies are supported at
             present (URLs of the form http://host[:port] or socks://[user:pass@]host[:port]).  No authentication is
             supported for HTTP.

Can proxy be set for dedicated connection only?

Tagging and regexp capturing brackets

Hi,

So I made a script and try to get its result with FDM:

match pipe "python3 -sqI /home/siltaar/.sbin/spam_test.py" returns (,'^(drop|keep)$') action tag "%0" continue
match tagged "keep" action maildir "%h/.Maildir/.spam"
match tagged "drop" action drop

But nothing seems to get tagged… Am I missing something obvious ?
(my script returns "drop" for spam to drop, "keep" for spam to investigate, or nothing for not spam, and I need these 3 states…).

How to debug error with google account?

Hello,
I configured fdm for two google accounts and one of them works fine (i.e. obvious without error) while the other one gives error:

gm-list: unexpected data: -ERR [AUTH] Username and password not accepted.

I have following in ~/.fdm.conf:

account "gm-list"   pop3s server "pop.gmail.com" port 995 user "lomov.vl"       pass "PASSWORD"   new-only cache $db + "-gl" keep
account "gm-sci"    pop3s server "pop.gmail.com" port 995 user "v.p.lomov"      pass "PASSWORD"   new-only cache $db + "-gs" keep

P.S. As I figured out accounts have to have different cache files when 'fcntl' lock-types is used, because other way I get "Bad file descriptor".

Certificate verification is disabled by default

Unless user adds "set verify-certificates", certificates are not checked at all. As fdm tries to be secure by default (for example, it checks file permissions), it would be good to always enable certificate verification. If certificate verification is not needed, it can be disabled with "no-verify" keyword. Otherwise, user may not even notice that no verification is performed.

Quick search for "fdm.conf" yields examples of configuration files that use TLS but don't enable certificate verification:
https://wiki.archlinux.org/index.php/fdm
https://gist.github.com/kgaughan/759186

new-only option must appear before keep

If an account is defined with keep and new-only options, they must be specified in a certain order. For example:

account "imap" imaps server "blah" user "user" pass "pass" keep new-only

This throws an error:

~ $ fdm fetch
/home/dave/.fdm.conf: syntax error at line 6

This works though:

account "imap" imaps server "blah" user "user" pass "pass" new-only keep

in gentoo linux, fdm can retrieve POP3 from yahoo but keep getting auth error with disroot.org

fdm -kvvv fetch working while retrieving pop3 from yahoo.com but throws auth error with pop3s disroot.org

Save attachment?

I would like to select certain mails and extract their attachments to a folder. Can fdm do it?

%n is not replaced with the user's ID

When I set lock-file "/run/user/%n/fdm.lock", I get this error:

/run/user//fdm.lock: open: Permission denied

According to the manual, %n should be replaced with "The delivery user's uid." But set lock-file "%h/.cache/fdm.lock" works as expected, so "The delivery user's home directory." is expanded...

Matching Regex doesn't always work when account type is a mbox in dos file

This issue troubled me quite a bit.

account "import" disabled mbox "%h/my_exported_gmail.mbox" keep
account "stdin" disabled stdin
# omitted actions

match "^From:.*abc" action "another.maildir"

This isn't working via import account however it is when I try to pipe in the single email via stdin account. The culprit is the Google takeout mbox file is in dos format with CRLF. After trim the CR out of the file, it's working like a charm.

Although it may not worth a fix, it's good to leave a note here for someone having the similar situation.

^(To|Cc|Bcc):(.,){20,}.$: regexec failed

Hi,

I'm working on another "spam" detection, trying to eliminate emails with more than 20 recipients using the following rule:
match "^(To|Cc|Bcc):(.*,){20,}.*$" in headers action maildir "%h/.Maildir/.spam"

For non-compliant messages it's ok, but a compliant message reports the error in subject of this message.

I tried the regexp with grep, it's not getting the message because all recipients are not trully in one line, or, if I try to get only one recipient it works, but I never get an error this way.

It remembers me that too long regexps (for instance : From: user@domain|user@domain… ) must sometimes be split in smaller ones. Here I try to match big things, it might be related…

Uninformative logging when smtp action lacks optional 'to'

With:
action "localdelivery" smtp server "localhost"
in ~/.fdm.conf and match all action "localdelivery", crontab -l == */12 * * * * /usr/local/bin/fdm fetch >/dev/null, logs fill up with:
datestamp hostname fdm: vfprintf %s NULL in "smtp%s server "%s" port %s to "%s""

This can be traced to deliver_smtp.c lines 235-237, function deliver_smtp_desc:

xsnprintf(buf, len, "smtp%s server "%s" port %s to "%s"",
data->server.ssl ? "s" : "", data->server.host, data->server.port,
data->to.str);

Modifying .fdm.conf to specify 'to "[email protected]"' avoids the null pointer. At a guess, it can also be avoided by modifying line 237 to data->to ? data.to.str : "" or something. Possibly it needs two different invocations, checking whether the optional to has been initialized. Possibly instead the default to should get initialized before invocation. Sorry that I can't be clearer, but it should be easily reproducible by setting up the action and rule above, with a target to pull from as well (without deleting from server, while testing (umm, sorry, you know these things, I'm sure)), and viewing logs. It's not a big deal, mostly cosmetic in logs, but getting log-splattered is always at least mildly annoying.

Can't login to imap server

$ fdm -k  -vv  fetch 
version is: fdm 1.7, started at: Sat Jan  7 11:13:42 2017
running on: Linux 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64
host is: Inspiron localhost 127.0.0.1
home is: /home/taoqy
loading configuration from /home/taoqy/.fdm.conf
added action "inbox": deliver=0:mbox "%h/mail/INBOX" 
added account "xx@xx": fetch=imaps server "imap.exmail.qq.com" port 993 user "xx@xx" folders "INBOX"
added rule 0: matches=all actions="inbox"
configuration loaded
locking using: flock 
options are: keep-all, verify-certificates, maximum-size=33554432, timeout=900, default-user="taoqy", command-user="taoqy", file-umask=077, queue-high=2, queue-low=1, strip-characters="\<>$%^&*|{}[]"'`;"
using tmp directory: /tmp
parent: 3 file descriptors in use
parent: started, pid is 25677
parent: 0 children, 0 dead children
parent: child 25678 (xx@xx) started
xx@xx: 4 file descriptors in use
xx@xx: fetch started, pid 25678
xx@xx: user is 1000
xx@xx: started processing
xx@xx: fetching
xx@xx: unexpected data: 2 BAD ȱ��login��������
xx@xx: fetching error. aborted
xx@xx: 0 messages processed in 5.117 seconds
xx@xx: polled for 0.261 seconds (0.000 blocked)
xx@xx: finished processing. exiting
parent: sending exit message to child 25678
parent: 1 children, 0 dead children
xx@xx: 4 file descriptors in use
parent: 1 children, 0 dead children
parent: child 25678 socket error
parent: 1 children, 0 dead children
parent: child 25678 returned 1
parent: finished, total time 5.118 seconds
parent: 3 file descriptors in use
host is: Inspiron localhost 127.0.0.1
home is: /home/taoqy
loading configuration from /home/taoqy/.fdm.conf
added action "inbox": deliver=0:mbox "%h/mail/INBOX" 
added account "xx@xx": fetch=imaps server "imap.exmail.qq.com" port 993 user "xx@xx" folders "INBOX"
added rule 0: matches=all actions="inbox"
configuration loaded
locking using: flock 
options are: keep-all, verify-certificates, maximum-size=33554432, timeout=900, default-user="taoqy", command-user="taoqy", file-umask=077, queue-high=2, queue-low=1, strip-characters="\<>$%^&*|{}[]"'`;"
using tmp directory: /tmp
parent: 3 file descriptors in use
parent: started, pid is 25677
parent: 0 children, 0 dead children
parent: child 25678 (xx@xx) started
xx@xx: 4 file descriptors in use
xx@xx: fetch started, pid 25678
xx@xx: user is 1000
xx@xx: started processing
xx@xx: fetching
xx@xx: unexpected data: 2 BAD ȱ��login��������
xx@xx: fetching error. aborted
xx@xx: 0 messages processed in 5.117 seconds
xx@xx: polled for 0.261 seconds (0.000 blocked)
xx@xx: finished processing. exiting
parent: sending exit message to child 25678
parent: 1 children, 0 dead children
xx@xx: 4 file descriptors in use
parent: 1 children, 0 dead children
parent: child 25678 socket error
parent: 1 children, 0 dead children
parent: child 25678 returned 1
parent: finished, total time 5.118 seconds
parent: 3 file descriptors in use

And successfully do this in python

import imaplib
mail = imaplib.IMAP4_SSL("imap.exmail.qq.com", 993)
mail.login("xx@xx", "passwd")
print mail.list()

Split multiple accounts into their own folder

Howdy! I'm trying to figure out how to split multiple accounts into their own folders on maildir like mail/riseup and mail/countermail.

segfault on fetching from specific pop server

I'm running the latest git master and getting a segfault when fetching mail from a specific pop server. Below is the output of fdm -a mana1 -vvv fetch. The mana1 account is the one that exhibits this segfault. All is fine on my other accounts.

version is: fdm 1.9, started at: Tue Jan 23 16:07:03 2018
running on: Linux 4.14.14-1-ARCH #1 SMP PREEMPT Sat Jan 20 08:49:12 UTC 2018 x86_64
host is: fugu fugu.vesath.org 0.0.0.0
home is: /home/bisson
loading configuration from /home/bisson/.fdm.conf
added account "stdin": fetch=stdin
added account "mochi": fetch=maildir "/mnt/mochi/home/bisson/Maildir"
added account "phare": fetch=imaps server "phare.normalesup.org" port imaps user "bisson" folders "INBOX"
added account "arch": fetch=imaps server "mail.archlinux.org" port imaps user "bisson" folders "INBOX"
added account "mana1": fetch=pop3 server "pop.mana.pf" port pop3 user "bisson"
added rule 0: matches=all lambda=0:maildir "~/var/sec/mail/spool" 
configuration loaded
locking using: flock 
options are: maximum-size=62914560, timeout=600, default-user="bisson", command-user="bisson", file-umask=077, queue-high=2, queue-low=1, strip-characters="\<>$%^&*|{}[]"'`;"
using tmp directory: /tmp
parent: started, pid is 8731
parent: 0 children, 0 dead children
parent: child 8733 (mana1) started
mana1: fetch started, pid 8733
mana1: user is 101
mana1: started processing
mana1: fetching
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9a1fc0, flags 0x02)
mana1: fetch, again
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9acef0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae170, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae3f0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae3f0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae4b0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae4b0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae560, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae560, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae9b0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae9b0, flags 0x02)
mana1: fetch, again
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9adbe0, flags 0x02)
mana1: fetch, again
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9aecc0, flags 0x02)
mana1: fetch, again
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9adfa0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae760, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ae760, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ad010, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ad010, flags 0x02)
mana1: fetch, again
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ad1a0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ad1a0, flags 0x02)
mana1: fetch, block
mana1: queued 0; blocked 0; flags 0x02
mana1: polling: 2, timeout=600000
mana1: fetch loop start
mana1: calling fetch state (0x555bbc9ad1a0, flags 0x02)
mana1: fetch, mail
mana1: found 47 lines, 27 in body
mana1: message-id is: <[email protected]>
parent: 1 children, 0 dead children
parent: child 8733 socket error
parent: 1 children, 0 dead children
parent: waiting for children
parent: child 8733 got signal 11
parent: waiting for children
parent: waiting for children
parent: finished, total time 1.201 seconds

No mail gets written and dmesg says:

fdm[8733]: segfault at 20 ip 00007f68730baf37 sp 00007ffe2e058298 error 4 in libc-2.26.so[7f6872f64000+1ae000]

Any idea how I could solve or help investigate this?

Fetching only new mail

I'm trying to run both a task that only fetches the new mail (via cron every 15 minutes) like:

fdm -kv -a [email protected]_new fetch

And one for all the mail every night:

fdm -kv -a [email protected] fetch

The problem is that the only solution I could come up with uses separate accounts using the new-only option e.g.:

account "[email protected]_new"
imaps server "mail.example.com"
port 993
user "user"
pass ...
new-only

This has the effect of giving my duplicate mail because the "Received: " header is different:

  --> diff 1593148327.17714_0.ostrich 1593135907.18272_0.ostrich

  3,4c3,4
  < Received: by ostrich (fdm 1.7, account "[email protected]");
  < 	Fri, 26 Jun 2020 01:12:07 -0400
  ---
  > Received: by ostrich (fdm 1.7, account "[email protected]_new");
  > 	Thu, 25 Jun 2020 21:45:07 -0400

What can I do?

I'm assuming there is a more elegant way to do this but after reading the man page I couldn't find it.

Using IMAP IDLE

I'd like to be able to be able to connect to an IMAP source to collect email, and register for new mail events using the IDLE command. The same suggested daemon-mode could also periodically poll, for those servers that don't support IDLE, I suppose; both are 'better' than a loop outside fdm that keeps on re-invoking it, and leading to all that authentication being done again (also it means that I could use fdm to sensibly prompt for credentials, rather than leave them on disk somewhere)

If this something that could be done?

Adding a header after rewrite unfolds other headers (under some unknown circumstances)

Using an action that rewrites the message, and then adds a header, as in

action "bogofilter" {
    remove-header "X-Bogosity"
    rewrite "/usr/local/bin/bogofilter -e -p -u"
    add-header "X-Filtered-Date" value "%[rfc822date]"
}

seems to sometimes unfold the headers in the message that is later saved to a mailbox. By "unfolding" I mean removes newlines, making each header line span only a single line.

Removing the add-header action from the above composite action resolves this and leaves the headers untouched.

I'm not entirely under what circumstances the headers are unfolded (I haven't been able to write a set of actions+rules to actually provoke this behaviour), but in all cases when this does occur, it happens consistently and removing the addition of a new header to the message seems to resolve it.

I'm using fdm 1.9 as packaged with OpenBSD-current, but this was also seen with fdm compiled from the master branch.

fetching with socks proxy bad protocol version: 1

Here is the output of fdm -kvv fetch

......
added rule 101: matches=all actions="inbox"
added account "gmail": fetch=pop3s server "pop.googlemail.com" port 995 user "user"
configuration loaded
using proxy: SOCKS5 on proxy.company.fr:1080
locking using: flock
options are: keep-all, maximum-size=20971520, timeout=900, default-user="rkm", command-user="rkm", file-umask=077, queue-high=2, queue-low=1, strip-characters="\<>$%^&*|{}[]"'`;"
using tmp directory: /tmp
parent: started, pid is 64218
parent: 0 children, 0 dead children
parent: child 90799 (gmail) started
gmail: fetch started, pid 90799
gmail: user is 1000
gmail: started processing
gmail: fetching
gmail: fetch loop start
gmail: calling fetch state (0x1955d8410c70, flags 0x02)
gmail: fetch, again
gmail: fetch loop start
gmail: calling fetch state (0x1955d841df10, flags 0x02)
gmail: bad protocol version: 1
gmail: fetch, error
gmail: fetching error. aborted
gmail: 0 messages processed in 0.086 seconds
gmail: finished processing. exiting
gmail: sending exit message to parent
gmail: waiting for exit message from parent
parent: got message type 1, id 0 from child 90799
parent: sending exit message to child 90799
parent: 1 children, 0 dead children
parent: 1 children, 0 dead children
parent: child 90799 socket error
parent: 1 children, 0 dead children
parent: waiting for children
parent: child 90799 returned 1
parent: waiting for children
parent: waiting for children
parent: finished, total time 0.088 seconds
.....

Error is triggered from here: https://github.com/nicm/fdm/blob/master/connect.c#L434

Support/Discussion mailing list/forum

Hello,
seems that sourceforge ML is down or closed so there is no dedicated ML for discussion or support. Is there any plan to create new ML? May be on google groups?

Fail to fetch mails having large attachments.

I'm using fdm to fetch emails from pop3 server. However I get 'message too big: 33554473 bytes' when some mail have a large attachment. Is it a limit from pop3 server? Or should I configure fdm to support this usage?

build failure with musl libc due to REG_STARTEND extension.

Build fails on Alpine Linux (with musl libc):

depbase=`echo re.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DPACKAGE_NAME=\"fdm\" -DPACKAGE_TARNAME=\"fdm\" -DPACKAGE_VERSION=\"2.0\" -DPACKAGE_STRING=\"fdm\ 2.0\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"fdm\" -DVERSION=\"2.0\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_TREE_H=1 -DHAVE_MREMAP=1 -DHAVE_SETRESUID=1 -DHAVE_SETRESGID=1 -DHAVE_STRLCPY=1 -DHAVE_STRLCAT=1 -I.   -Os -fomit-frame-pointer -I/usr/local/include -DSYSCONFFILE="\"/etc/fdm.conf\"" -DSYSLOCKFILE="\"/var/run/fdm.lock\""   -Os -fomit-frame-pointer -D_GNU_SOURCE -std=gnu99 -O2  -MT re.o -MD -MP -MF $depbase.Tpo -c -o re.o re.c &&\
mv -f $depbase.Tpo $depbase.Po
In file included from fdm.h:23,
                 from re.c:25:
/usr/include/sys/cdefs.h:1:2: warning: #warning usage of non-standard #include <sys/cdefs.h> is deprecated [-Wcpp]
    1 | #warning usage of non-standard #include <sys/cdefs.h> is deprecated
      |  ^~~~~~~
In file included from re.c:25:
fdm.h:728:1: warning: parameter names (without types) in function declaration
  728 | int b64_ntop(src, srclength, target, targsize);
      | ^~~
fdm.h:729:1: warning: parameter names (without types) in function declaration
  729 | int b64_pton(src, target, targsize);
      | ^~~
re.c: In function 're_block':
re.c:85:43: error: 'REG_STARTEND' undeclared (first use in this function); did you mean 'REG_EPAREN'?
   85 |  res = regexec(&re->re, buf, NPMATCH, pm, REG_STARTEND);
      |                                           ^~~~~~~~~~~~
      |                                           REG_EPAREN
re.c:85:43: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [Makefile:657: re.o] Error 1

To my understanding, REG_STARTEND is not specified in POSIX.

Build fails on Kiss linux

Hello,

I'm trying to build fdm on Kiss linux without success. Here is what I do:

./configure --prefix=/usr

make

I get after the make:

In file included from re.c:25:
fdm.h:728:1: warning: parameter names (without types) in function declaration
  728 | int b64_ntop(src, srclength, target, targsize);
      | ^~~
fdm.h:729:1: warning: parameter names (without types) in function declaration
  729 | int b64_pton(src, target, targsize);
      | ^~~
re.c: In function ‘re_block’:
re.c:85:43: error: ‘REG_STARTEND’ undeclared (first use in this function); did you mean ‘REG_EPAREN’?
   85 |  res = regexec(&re->re, buf, NPMATCH, pm, REG_STARTEND);
      |                                           ^~~~~~~~~~~~
      |                                           REG_EPAREN
re.c:85:43: note: each undeclared identifier is reported only once for each function it appears in
make[1]: *** [Makefile:657: re.o] Error 1
make[1]: *** Waiting for unfinished jobs....
In file included from shm-mmap.c:26:
fdm.h:728:1: warning: parameter names (without types) in function declaration
  728 | int b64_ntop(src, srclength, target, targsize);
      | ^~~
fdm.h:729:1: warning: parameter names (without types) in function declaration
  729 | int b64_pton(src, target, targsize);
      | ^~~
In file included from strb.c:26:
fdm.h:728:1: warning: parameter names (without types) in function declaration
  728 | int b64_ntop(src, srclength, target, targsize);
      | ^~~
fdm.h:729:1: warning: parameter names (without types) in function declaration
  729 | int b64_pton(src, target, targsize);
      | ^~~
In file included from timer.c:25:
fdm.h:728:1: warning: parameter names (without types) in function declaration
  728 | int b64_ntop(src, srclength, target, targsize);
      | ^~~
fdm.h:729:1: warning: parameter names (without types) in function declaration
  729 | int b64_pton(src, target, targsize);
      | ^~~
make[1]: Leaving directory '/home/user/.cache/kiss/sources/fdm/fdm-2.0'
make: *** [Makefile:473: all] Error 2

I don't know if it can help, on Kiss linux libressl is used and musl.

Do you have any idea what can the problem ? Thank you!

Unable to fetch mail from Apple mail server

Configuration

action "inbox" maildir "%h/Mail/INBOX"

account "jimmyoo" imaps server "imap.mail.me.com" port 993 user "[email protected]" pass "<app-specific-passowrd>"

match all action "inbox"

command output

➜ fdm -vvvv fetch
version is: fdm 2.0, started at: Fri Jan 17 10:08:59 2020
running on: Linux 5.4.11-arch1-1 #1 SMP PREEMPT Sun, 12 Jan 2020 12:15:27 +0000 x86_64
host is: archlinux-thinkpad archlinux-thinkpad.localdomain 127.0.1.1
home is: /home/jinmiaoluo
loading configuration from /home/jinmiaoluo/.fdm.conf
added action "inbox": deliver=0:maildir "%h/Mail/INBOX"
added account "jimmyoo": fetch=imaps server "imap.mail.me.com" port 993 user "[email protected]" folders "INBOX"
added rule 0: matches=all actions="inbox"
configuration loaded
locking using: flock
options are: maximum-size=33554432, timeout=900, default-user="jinmiaoluo", command-user="jinmiaoluo", file-umask=077, queue-high=2, queue-low=1, strip-characters="\<>$%^&*|{}[]"'`;"
using tmp directory: /tmp
parent: started, pid is 592626
parent: 0 children, 0 dead children
parent: child 592629 (jimmyoo) started
jimmyoo: fetch started, pid 592629
jimmyoo: user is 1000
jimmyoo: started processing
jimmyoo: fetching
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce9732a0, flags 0x02)
jimmyoo: fetch, again
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce978440, flags 0x02)
jimmyoo: fetch, block
jimmyoo: queued 0; blocked 0; flags 0x02
jimmyoo: polling: 2, timeout=900000
< * OK [CAPABILITY mr36p00im-tygg11191101 1913B48 XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR AUTH=ATOKEN AUTH=PLAIN] iSCREAM ready to rumble (1913B48-10000-1913B-0-9cb09718163b:10813) mr36p00im-tygg11191101 [33:111:02:08:55:D8]
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce978ba0, flags 0x02)
jimmyoo: fetch, block
jimmyoo: queued 0; blocked 0; flags 0x02
jimmyoo: polling: 2, timeout=900000
> 1 CAPABILITY
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce978cb0, flags 0x02)
jimmyoo: fetch, block
jimmyoo: queued 0; blocked 0; flags 0x02
jimmyoo: polling: 2, timeout=900000
< * CAPABILITY mr36p00im-tygg11191101 1913B48 XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR AUTH=ATOKEN AUTH=PLAIN
1 OK !!
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce978cb0, flags 0x02)
jimmyoo: fetch, again
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce97a6e0, flags 0x02)
jimmyoo: fetch, block
jimmyoo: queued 0; blocked 0; flags 0x02
jimmyoo: polling: 2, timeout=900000
> 2 LOGIN {18}
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce978e20, flags 0x02)
jimmyoo: fetch, block
jimmyoo: queued 0; blocked 0; flags 0x02
jimmyoo: polling: 2, timeout=900000
jimmyoo: fetch loop start
jimmyoo: calling fetch state (0x5588ce978e20, flags 0x02)
jimmyoo: fetch, block
jimmyoo: queued 0; blocked 0; flags 0x02
jimmyoo: polling: 2, timeout=900000
jimmyoo: connection closed
jimmyoo: fetching error. aborted
jimmyoo: 0 messages processed in 1.320 seconds
jimmyoo: finished processing. exiting
jimmyoo: sending exit message to parent
jimmyoo: waiting for exit message from parent
parent: got message type 1, id 0 from child 592629
parent: sending exit message to child 592629
parent: 1 children, 0 dead children
parent: 1 children, 0 dead children
parent: child 592629 socket error
parent: 1 children, 0 dead children
parent: waiting for children
parent: child 592629 returned 1
parent: waiting for children
parent: waiting for children
parent: finished, total time 1.322 seconds

No smtp authentication?

Just to be sure, could anybody tell me if FDM support SMTP authentication? (when sending email via SMTP of course). If it doesn't as I'm afraid of, would you use an external app or otherwise?
Thank you!

REGRESSION: IMAPS message count calculation incorrect

I am testing a setup involving a local server and an automatic fetch system, which I am fetching:

Folder "INBOX" has 2 messages
Folder "Sent" has 1 message

This totals 3 messages, and yet when I poll:

$ fdm poll
main: 5 messages found

I then added some extra lines in the config file to see if I could find the issue.
My current test ~/.fd.conf is now:

set no-received
set maximum-size 128M

$au = "foo"
$ap = "bar"
$as = "192.168.1.99"

account "main" imaps server "${as}" user "${au}" pass "${ap}" folders { "INBOX" "Sent" }

account "catchall" imaps server "${as}" user "${au}" pass "${ap}"
account "main-inbox" imaps server "${as}" user "${au}" pass "${ap}" folder "INBOX"
account "main-sent" imaps server "${as}" user "${au}" pass "${ap}" folder "Sent"

match all action pipe "/usr/local/libexec/dovecot/dovecot-lda -e"

The "catchall", "main-inbox", and "main-sent" accounts were added to see if it gets individual folders wrong.

When polling:

$ fdm poll
main-inbox: 2 messages found
main-sent: 1 messages found
catchall: 2 messages found
main: 5 messages found

If I delete just one message from the INBOX, the total reported is now 3. If I then delete the other message from the INBOX, the total reported is 1, which is correct (since it counted the message in the Sent folder).

fdm reports correctly with multiple messages when only one folder contains all the messages.

As per above, there are 3 messages to fetch from. On one run in a production system, it claims to have found 116 messages on the server, yet there were only around 30 when I checked and added up all messages in the folders. This occurs often, where the "messages found" is a number at least 3-5 times the actual number of messages present.

This bug appears to result in the "folders { ... }" syntax being unusable, as the program hangs after fetching just a few of the mails. Some sample output with 6 messages in the system (fdm poll reports 9):

...
main: message 6 delivered (rule 0, pipe) in 0.152 seconds
main: calling fetch state (0x7379e21a470, flags 0x00)
main: fetch, block
main: queued 1; blocked 0; flags 0x00
parent: 2 children, 5 dead children
main: polling: 2, timeout=0
main: fetch loop start
main: trying (deliver) message 6
main: deliver message 6, match
parent: child 1105 socket error
main: calling fetch state (0x7379e21a470, flags 0x00)
main: fetch, block
main: queued 1; blocked 0; flags 0x00
main: polling: 2, timeout=0
parent: 2 children, 5 dead children
main: fetch loop start
main: trying (match) message 6
main: match message 6, done
main: deleting message 6
main: calling fetch state (0x7379e21a470, flags 0x02)
main: fetch, block
main: queued 0; blocked 0; flags 0x02
main: polling: 2, timeout=900000
main: fetch loop start
main: calling fetch state (0x7379e219800, flags 0x02)
main: fetch, block
main: queued 0; blocked 0; flags 0x02
main: polling: 2, timeout=900000
parent: waiting for children
parent: child 1105 returned 0
parent: waiting for children
parent: 1 children, 6 dead children
main: fetch loop start
main: calling fetch state (0x7379e219800, flags 0x02)
main: fetch, again
main: fetch loop start
main: calling fetch state (0x7379e21a470, flags 0x02)
main: fetch, block
main: queued 0; blocked 0; flags 0x02
main: polling: 2, timeout=900000

It hangs at the very end, and 'top' reveals that two fdm processes are present, stuck on WAIT state "poll".

Using OpenBSD 5.9 (GENERIC.MP), with fdm-1.9 package (pkg_add). This appears to be a regression that was once fixed in 2009, according to the CHANGES file.

HOME seems to get lost somewhere

hello,

It seems that maildir delivery doesn't honor the '-h' option as the home gets reset back to the passwd home somewhere in the code path.

Below is a sample verbose log of me reproducing the issue:

$ export HOME=/tmp2; fdm -vv -h -m -a mda fetch < /etc/myname 
version is: fdm 1.9, started at: Tue Sep 17 09:52:46 2019
running on: OpenBSD 6.5 GENERIC.MP#1 amd64
host is: mailbrix.mx mailbrix.mx 95.179.171.116
home is: /tmp2
loading configuration from /etc/fdm.conf
[...]
mda: deliver started, pid 88708
mda: deliver user is: _vusers (1002/1002), home is: /var/maildir
mda: saving to maildir /var/maildir/
mda: creating /var
mda: creating /var/maildir//cur
mda: creating /var/maildir//new
mda: creating /var/maildir//tmp
mda: writing to /var/maildir//tmp/1568713966.88708_0.mailbrix.mx
mda: moving .../tmp/1568713966.88708_0.mailbrix.mx to .../new/1568713966.88708_0.mailbrix.mx
mda: message 1 delivered (rule 3, maildir) in 0.015 seconds
parent: sending exit message to child 88708
mda: deleting message 1
parent: child 88708 returned 0
parent: 1 children, 1 dead children
mda: 1 messages processed (0 kept) in 0.020 seconds (average 0.020)
mda: finished processing. exiting
parent: sending exit message to child 14033
parent: child 14033 returned 0
parent: finished, total time 0.023 seconds

message too big: 10485807 bytes aborted.

I have error saying message too big as goes the title of this bug report. and it says "fetching error. aborted".

How do I workaround this issue. Can I skip this big message and continue to fetch remaining mails ?

Unable to build latest revision on OpenBSD 6.1

In the latest revision (cae4ea3) I cannot run configure on OpenBSD 6.1.

$ automake-1.15
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/${ <-- HERE ([^ \t=:+{}]+)}/ at /usr/local/bin/automake-1.15 line 3936.
Provide an AUTOCONF_VERSION environment variable, please
automake-1.15: error: autoconf failed with exit status: 127
$ autoconf-2.69
configure.ac:8: error: possibly undefined macro: AM_INIT_AUTOMAKE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:18: error: possibly undefined macro: AM_PROG_CC_C_O
configure.ac:30: error: possibly undefined macro: AM_CONDITIONAL
$ ./configure
configure: error: cannot find install-sh, install.sh, or shtool in etc "."/etc

What am I doing wrong? This worked flawlessly on OpenBSD 5.9/6.0, but has stopped working as of 6.1.

Next release?

I have noticed that this project gets regular commits, however the last release was v1.9 which dates back in 2015.
Is there any plans for a new release (looking particularly at an updated OpenBSD package).

fdm fails to fetch mails from some pop3 server

Hi, I can see those mails printed in the terminal but it constantly hits parent: child 25183 socket error which aborts current server fetching. How can I make fdm retry or at least put those mails that have certainly been fetched into maildir?

Sync local changes back to server

I have an Outlook email address that works very slowly over IMAP. It's
practically unusable with Mutt's IMAP feature.

It works much better if I fetch the emails with fdm and use Mutt to manage
them locally. The problem with that, however, is any operations performed on
the emails - such as deleting or moving them to another folder - aren't synced
with the server. Then, when I use Mutt on another computer with the same
account, I have to do it all over again.

Can fdm sync local changes back to the IMAP server so the folders will be
mirrored?

Can't connect to google apps account using IMAP

Trying fdm on archlinux, but I can't get my mail through IMAP. This is my .fdm.conf

action "inbox" maildir "%h/Mail/INBOX"
account "mycompany" imaps server "imap.gmail.com" port 993 user "[email protected]" pass
"password"
match all action "inbox"

When I try fdm -kv fetch I get this error:

unexpected data: 2 NO [ALERT] Please log in via your web browser: https://support.google.com/mail/accounts/answer/78754 (Failure)

IMAP is enabled on the account.

Can fdm send mail?

I noticed in the manual "An action may be defined to pass mail on over SMTP." Does this mean fdm can be used to send mail? How would I use the following action?

action "mysmtpaction" smtp server "smtp.server"

Better CHANGES please?

Hi,

Is there any nice reference for changelog? The only file there seems dated for years.

Thanks.

can i avoid storing passwords in plain text?

is there no way to avoid storing passwords in plain text?

dispaly delivered directory name in verbose

I have message like below from fdm -v -k
msg: got message 2 of 2: size 994, body 973
msg: message 2 delivered (rule 82, maildir) in 0.019 seconds

How do i find whicg maidir directory this mail was delivered. I had a lot of filters in my fdm.conf

for example

...
$ubuntu = "${path}/Ubuntu"
...
action "ubuntu" maildir "${ubuntu}/News"
...

now if fdm fetch a mail from sever that matches exactly ubuntu/News. How do I know that information in verbose.? Is there already an option for that?

Encrypted password.

Hi.

Is it possible to store the account password encrypted, or make pass run gpg to decrypt and get password from a file?

For instance, with isync it is possible to use something like this:

PassCmd "gpg2 -q --for-your-eyes-only --no-tty -d ~/somefile.gpg"

Thanks.

strlcat misdetection with libressl

Portable libressl contains symbols for strlcat/strlcpy, but no prototypes, resulting in possibly invalid calls to these functions. I think instead of AC_CHECK_FUNC, AC_CHECK_DECLS should be used instead.

SMTP - FROM & TO

I'm currently using the FDM as a relay between POP3 and SMTP (Postfix). My Postfix mailserver is currently rejecting all the received mail from FDM due to incorrect from and to e-mail headers. Once it receives the mail processed by FDM the addresses seem different:

Config:

action "tosmtp" smtp server "mail.mydomain.eu"
account "pop3" pop3 server "mail.source.eu" user <user> pass <pass>
match all action "tosmtp"

Mail:

FROM: [email protected]
TO: [email protected]

Result at the mailserver "mail.mydomain.eu":

FROM: [email protected]
TO: [email protected]

If I add a forced "to" in the action, it gets delivered:

action "tosmtp" smtp server "mail.mydomain.eu" to "[email protected]"

The header information in the mailbox is (when "to" is set):

Return-Path: <[email protected]>
Delivered-To: [email protected]
Received: from mail.mydomain.eu (localhost [127.0.0.1])
       by mail.mydomain.eu (Postfix) with ESMTP id 74DDC1C1143
       for <[email protected]>; Fri, 26 Aug 2016 15:05:33 +0200 (CEST)
...
From: <[email protected]>
To: <[email protected]>
...

Postfix rejection, "/var/postfix/mail.log"

Aug 26 14:40:29 mydomain postfix/smtpd[5641]: connect from localhost[127.0.0.1]
Aug 26 14:40:29 mydomain postfix/smtpd[5641]: 709091C1143: client=localhost[127.0.0.1]
Aug 26 14:40:29 mydomain postfix/cleanup[5637]: 709091C1143: message-id=<[email protected]>
Aug 26 14:40:29 mydomain postfix/qmgr[4898]: 709091C1143: from=<[email protected]>, size=4025, nrcpt=1 (queue active)
Aug 26 14:40:29 mydomain postfix/smtpd[5641]: disconnect from localhost[127.0.0.1]
Aug 26 14:40:29 mydomain amavis[1542]: (01542-03) Passed CLEAN {RelayedInternal}, MYNETS LOCAL [127.0.0.1]:44098 [77.171.48.199]  -> , Queue-ID: 1E66F1C1141, Message-ID: , mail_id: Ok1rTk5Ld4hm, Hits: 6.294, size: 3167, queued_as: 709091C1143, 333 ms, Tests: [DOS_OUTLOOK_TO_MX=1.449,FSL_HELO_NON_FQDN_1=0.001,HTML_MESSAGE=0.001,MIME_HTML_MOSTLY=0.001,RCVD_IN_PBL=3.558,RDNS_NONE=1.274,T_HEADER_FROM_DIFFERENT_DOMAINS=0.01]
Aug 26 14:40:29 mydomain postfix/local[5642]: 709091C1143: to=<[email protected]>, relay=local, delay=0.03, delays=0.01/0.01/0/0.02, dsn=2.0.0, status=sent (delivered to mailbox)

The last line should have said: to = [email protected], which works if "to" is set in the smtp action.

I've reviewed the code but to no avail.

Why and how are the FROM and TO headers manipulated?

Option to unwrap wrapped lines at 76 characters when saved in maildir

When emails are saved in maildirs, the text file is wrapped at 76 characters. Since the wrap could break between words, it affects searching for keywords across emails. An option to unwrap automatically for text like content types would be helpful.

regex for chinese only subjects (working with grep -G, not with fdm)

Hi,

I tried this config to trash chinese spam in a non-chinese mailbox :

account "a" maildir "~/.Maildir/
match not "^Subject: .*[a-zA-Z0-9[:punct:]]" in headers action maildir "%h/.Maildir/.spam"       
match all action keep

With grep it works like a cham:

$ echo "Subject: 家有小奴各种制服来诱 惑" | grep -G "^Subject: .*[a-zA-Z0-9[:punct:]]"
$
$ echo "Subject: 家有小奴各种制服来诱 惑à" | grep -G "^Subject: .*[a-zA-Z0-9[:punct:]]"
Subject: 家有小奴各种制服来诱 惑à
$

With FDM the message is kept in the inbox (when I apply the rule with the previous config as a filter), and worse, one legit message is spamed (among around a hundred others which are kept as expected):
Subject: Urssaf en ligne : Confirmation de votre adresse courriel

Here are some relevant -vvvv log line about the false positiv if it can help:

parent: 1 children, 2 dead children
a: reading mail from: /home/<user>/.Maildir//new/1489354359.6840_5.tabr
a: read 3903 bytes
a: fetch, mail   
a: found 73 lines, 33 in body
a: message-id is: <[email protected]>
a: found 20 wrapped lines
a: got message 69: size 3978, body 1887
a: fetch loop start
a: trying (match) message 69
a: matching from 0 to 1887 (size=3978, body=1887)
a: tried regexp "^Subject: .*[a-zA-Z0-9[:punct:]]" in headers, result now 1
a: finished rule 0, result 1
a: matched to rule 0
a: action <rule 0>:0 (maildir), user <user>
a: match message 69, deliver
a: trying (deliver) message 68
a: deliver message 68, blocked
a: trying (deliver) message 69
a: message 69, running action <rule 0>:0 (maildir) as user <user>
a: sending action to parent
a: deliver message 69, blocked
a: queued 2; blocked 2; flags 0x00
a: polling: 1, timeout=900000
parent: got message type 0, id 69 from child 8563
parent: deliver child 8566 started (uid 1000)
parent: 2 children, 2 dead children

Certificate fingerprint support

If server uses self-signed certificate, it is impossible to connect securely with fdm. There are two options to connect securely to the server that uses self-signed certificate: by adding CA to certificate bundle and by storing fingerprint (SSH-like trust on first use policy). The second option may be desirable when CA certificate is not available.

Please add a --dry-run option

It would be useful to have a --dry-run option to be able to validate that a rule-set is performing the intended action without actually altering a mailspool.
For every triggered match a message should be printed, stating what would happen if fdm is run without --dry-run.

Use variables in commands

Using variables in commands should be supported. I've tried both

$my_password = $(gpg --quiet --batch --use-agent --decrypt --output - "%h/path/to/my/password.gpg")

and

$my_password_file = "%h/path/to/my/password.gpg"
$my_password = $(gpg --quiet --batch --use-agent --decrypt --output - ${my_password_file})

but both result in the string being interpreted literally as a command.

nicm / fdm Goto Github PK

fdm's Introduction

fdm's People

Contributors

Stargazers

Watchers

Forkers

fdm's Issues

Configuration

command output

Recommend Projects

Recommend Topics

Recommend Org

Jobs