correctly - logically complete that now by removing MLINKS from base;
authors need only to ensure there is an entry in NAME for any function/
util being added. MLINKS will still work, and remain for perl to ease
upgrades;
ok nicm (curses) bcook (ssl)
ok schwarze, who provided a lot of feedback and assistance
ok tb natano jung
the enqueuer will revoke group and regain real gid right after mkstemp.
this would have prevented the symlink/hardlink attacks against offline, and
it will avoid having to deal with new ways users can mess with it.
ok eric@, ok millert@
- cleanup the internal queue backend API and get rid of the QOP_* thing.
- implement a queue_proc backend
- rename queue_fsqueue.c to queue_fs
- enable support for queue encryption
- add an envelope cache
- better logging and error reporting
* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright
FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.
smtpd.conf(5) simplified, it will require adaptations
ok eric@
- remove support for encrypted queue, it will be reintroduced later after
pouring more thinking into it
if you had it enabled, flush your queue before updating
using the new API. By default, OpenSMTPD does not provide queue encryption,
but it can be enabled with "queue encryption [args]" and will transparently
encrypt/decrypt envelopes/messages as they hit the queue.
By default, it will use Blowfish in CBC mode with a different random IV for
each envelope and message. User provided key is expanded using sha256 but a
different cipher and digest may be specified in smtpd.conf
Queue encryption is compatible with compression and if both options are set
it will do them in correct order and transparently.
tested by chl@, a few users and myself
ok chl@ and I
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure
ok eric@, ok chl@
couple of load/dump functions to convert to and from a human readable fmt.
while at it kill struct delivery and merge back its fields to the envelope.
this basically means we shouldn't require users to flush their queues every
time we make a change to struct envelope.
work is not done, but we're at a better state than the binary fsqueue so
we'll improve it in-tree.
has been running on my own box for the last 12 hours or so
ok eric@, chl@
- fsqueue->setup() performs the queue initialization;
- fsqueue->message() controls messages;
- fsqueue->envelope() controls envelopes;
This commit brings the following to fsbackend:
fsqueue_setup(), fsqueue_message_delete(), fsqueue_envelope_load(),
fsqueue_envelope_update(), fsqueue_envelope_delete().
It also makes smtpd use the queue_backend API for these operations.
is way more optimized than what we had earlier and there's definitely stuff
we want to keep, however it is early optimization that doesn't account for
many features and makes them hard (if not impossible) to write without
ugly workarounds that ruin the purpose of the optimizations.
the backout goes to 30 May's right before the commit and catches up on all
the non-queue related commits that happened since then.
i'll work on reintroducing the ideas from this queue when the basic
features we expect from a MTA are implemented.
suggested on tech@ about a week ago, no objections, several "please make
smtpd move forward" mails from hackers and tech readers.
Major goals:
1) Fix bad performance caused by the runner process doing full queue
read in 1s intervals. My Soekris can now happily accept >50 msg/s
while having multi-thousand queue; before, one hundred queue would
bring the system to its knees.
2) Introduce Qmail-like scheduler that doesn't write as much to the
disk so that it needs less code for servicing error conditions,
which in some places can be tricky to get right.
3) Introduce separation between the scheduler and the backend; these
two queue aspects shouldn't be too tied too each other. This means
that eg. storing queue in SQL requires rewrite of just queue_backend.c.
4) Make on-disk queue format architecture independent, and more
easily extensible, to reduce number of flag days in the future.
Minor goals:
ENOSPC no longer prevents delivery attempts, fixed session limiting
for relayed mail, improved batching of "relay via" mails, human-readable
mailq output, "show queue raw" command, clearer logging, sending
of single bounce about multiple recipients, exact delay= computation,
zero delay between deliveries while within session limit (currently
1s delay between re-scheduling is enforced), mta no longer requests
content fd, corrected session limit for bounce submissions, tiny
<100B queue files instead of multi-KB, detect loops before accepting
mail, reduce traffic on imsg channels by killing enormous struct
submit_status.
This restructures the client_* API internals significantly. The code becomes
pipelining in nature. All SMTP commands are put on the output queue and
dequeued as quickly as possible. Once dequeued, they're moved to the receive
queue so that replies can be matched with previous commands.
Dequeuing commands from the output queue halts when the count of commands
currently in-pipeline (``cmdi'') is equal to the command send window (``cmdw'').
There are three cmdw values useful in practice:
0 clear pipeline, ie. inhibit all future sends
1 disable pipelining, ie. use old ``one-request-one-reply`` mode
SIZE_T_MAX enable pipelining, ie. dequeue as many commands as possible
At the beginning of session cmdw is 1. When it is found that peer supports
PIPELINING, it grows to SIZE_T_MAX. After dequeing DATA it is again 1. After
sending QUIT it is 0.
Each command dequeued from the output queue becomes a buf in a msgbuf. The act
of combining multiple commands into a single send operation did not need to be
implemented: buf_write() already combines bufs using iovec and sends them at
once using sendmsg(2).
Tested by todd@ and oga@
"looks good" to gilles@
Make bounce code and /usr/sbin/sendmail interface use this new API.
The mta process continues to use its own implementation, but
eventually will be switched to use this shared module.
Buffer routines are taken from buffer.c rather than from evbuffer.
This is one step forward to using a single buffer API across the
program.
"it looks sexy" gilles@
without "talking" smtp to listeners. currently, a big part of the server
side code is done (and requires a cleanup), next step is to get it usable
properly from a mail user agent.
`showqueue' which displays the content of the queue (all envelopes)
`showrunqueue` which displays envelopes scheduled for delivery. The
utility will be improved and extended, but for now we need at least
this basic support to help debug queue-related issues.
Output format is spamdb-alike:
type|envelope uid|sender|recipient|last delivery date|retry count
ok jacek@
just an empty clone of relayctl with the glue needed to have it
exchange imsg with smtpd correctly. code mostly by pyr@, reviewed
by chl@ and I a while ago.
