camdd: Fix the worker startup race

The reader can hand off its first buffer before the writer sets
CAMDD_DEV_FLAG_ACTIVE.  camdd_queue_peer_buf() then mistakes the writer
for an exited peer and releases the buffer instead of queueing it.  This
can make a copy fail with bytes read but nothing written, without an
error message.

Mark both devices active before creating either worker, leaving the
workers to clear the flag on exit.  The work queues and EVFILT_USER
events are already initialized, so queued buffers and their notification
can wait for the writer to start without a startup barrier.

Treat pthread_create() failure as fatal.  Returning through the normal
cleanup path can free devices that an already-started worker is still
using, and an unstarted peer cannot consume any queued buffers.

Reviewed by:	imp, markj, ken
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D61277

usr.sbin/camdd/camdd.c

@@ -1684,8 +1684,6 @@ camdd_worker(void *arg)
 
 	pthread_mutex_lock(&dev->mutex);
 
-	dev->flags |= CAMDD_DEV_FLAG_ACTIVE;
-
 	for (;;) {
 		struct kevent ke;
 		int retval = 0;
@@ -3343,13 +3341,18 @@ camdd_rw(struct camdd_io_opts *io_opts, camdd_argmask arglist, int num_io_opts,
 		goto bailout;
 	}
 
+	/*
+	 * Mark both devices active before starting either worker so that the
+	 * reader can queue buffers before the writer starts running.
+	 */
+	for (i = 0; i < num_io_opts; i++)
+		devs[i]->flags |= CAMDD_DEV_FLAG_ACTIVE;
+
 	for (i = 0; i < num_io_opts; i++) {
 		error = pthread_create(&threads[i], NULL, camdd_worker,
 				       (void *)devs[i]);
-		if (error != 0) {
-			warnc(error, "pthread_create() failed");
-			goto bailout;
-		}
+		if (error != 0)
+			errc(1, error, "pthread_create() failed");
 	}
 
 	for (;;) {