[cpp-threads] N2800's C/C++MM and read-write locking

[Paul E. McKenney]

On Thu, Jan 08, 2009 at 04:01:28PM +0200, Peter Dimov wrote:
> >> I also often tend to think of "rwlocks" as shared/exclusive locks.  If 
> >> you really
> >> want to use weaker fences for readers, I think you would need to split 
> >> them
> >> into two variantes: shared/exclusive and read/write.
> >
> > Care to elaborate?
> 
> Something like:
> 
> take shared lock
> try STM transaction in a lock-free manner
> unlock
> 
> if failed
>     take write lock
>     use ordinary ops
>     unlock
> 
> Here the lock is not used as a pure read lock since the transaction contains 
> writes. 

I agree with Peter.  In fact, I have come across situations where one
read-acquires the lock to protect an update and write-acquires the lock
to protect a read.  What, you don't believe me?  ;-)  Read on, then...

Here is such an example, which assumes per-thread variables that I
represent as arrays, as I cannot remember what if anything C++0x does
with thread-local storage.  I also invent a me() function that returns
a small integer uniquely identifying the current thread, and I steal
the Linux-kernel syntax for reader-writer locks.

Yes, this sort of thing really has been used in production in real life.
Explanation after example.

	long count[N_THREADS] = 0;
	DEFINE_RWLOCK(rwl);

	void get_reference(void)
	{
		read_lock(&rwl);
		count[me()]++;
		read_unlock(&rwl);
	}

	void put_reference(void)
	{
		read_lock(&rwl);
		count[me()]--;
		read_unlock(&rwl);
	}

	/*
	 * Caller must write-hold rwl.
	 */
	int no_references(void)
	{
		int i;

		for (i = 0; i < N_THREADS; i++)
			if (count[i] != 0)
				return 0;
		return 1;
	}

	/* Sample use. */

	write_lock(&rwl);
	if (no_references())
		perform_action_requiring_there_be_no_references();
	write_unlock(&rwl);

One place I have seen this used is for hot-plugging disk drives.
When starting a disk read or write, one invokes get_reference().
When that read/write completes, one invokes put_reference().  When one
wants to remove a disk from service, one uses no_references() to test
whether there is an outstanding read or write to a disk.  One normally
has some mechanism to prevent new reads or writes from starting on a
given volume, and one normally also has a separate instance of count[]
for each disk drive.

In this case, the read-side primatives really do need heavy memory
fences, as the "readers" are updating the data structure.

						Thanx, Paul