[rds-devel] BUG_ON fired in rds

[Zach Brown]

> > I'll be happy to learn more on the problem&&  its solution, the patch
> > isn't documented...

See fs/bio.c:bio_{set,check}_pages_dirty().  O_DIRECT file IO has this exact
same problem.

> avoid the call in the trace Eli posted. The patch I posted marks pages 
> that will be the recipients of RDMA ops as dirty beforehand (from 
> process context), instead of right before the pages are unpinned.

The problem is that background writeback can find those dirty pages and clean
them before the RDMA actually writes to the pages.   We need another pass which
tests that the pages are still dirty.  If they aren't then they have to be
marked dirty again.

We probably also want to mirror its PageCompound() tests.  I smell an
opportunity for a VM helper function.

bio_check_pages_dirty() performs that post-io redirtying with scheduled work.
That'd be a performance problem if that's the only way the pages were dirtied.
But this path only hits if writeback cleans the pages while the IO is in
flight, which should be a rare case.

While we're in here, it'd be nice to finally put this comment out of its
misery:

	/* FIXME we need a way to tell a r/w MR
	 * from a r/o MR */

It'd be nice to record in the op whether RDMA is going to modify the page
contents.  Getting all the locking and processing involved in page dirtying out
of paths that don't need it would be really nice.

- z