[BDB 11gR2 Beta] Using FTS2 with the BDB SQLite compatibility library code [#18385]
Greg Burd
GREG.BURD at oracle.com
Wed Mar 3 07:53:30 PST 2010
Interesting... I wonder if the "design flaw" applies when BDB is VACUUM'ed as well... maybe we're immune to this issue.
-greg
> -----Original Message-----
> From: Sandra Whitman
> Sent: Wednesday, March 03, 2010 10:35 AM
> To: Greg Burd
> Cc: bdb-beta-feedback at oss.oracle.com; Hexxeh
> Subject: Re: [BDB 11gR2 Beta] Using FTS2 with the BDB SQLite
> compatibility library code [#18385]
>
> Thanks,
>
>
> I have made some of those changes, but not all. I will do that. Right
> now I am trying to add in the fts2 manual build. I'll let you know how
> that goes.
>
>
> One thing Emily told me last evening is that although the SQLite
> documentation version 3.5 is not compatible with fts2, from:
> http://www.mail-archive.com/sqlite-users@sqlite.org/msg31627.html,
> fts2 has a design flaw that would cause corruptions of fts2 index when
> running VACUUM against db. However if the VACUUM is not run, it should
> be fine.
>
>
> Thanks,
> Sandra
>
>
> Greg Burd wrote:
> > Sandra,
> >
> > sqlite-poison.patch fails because it changes the btree.c code which
> is not a shared part of the BDB version. We use that file to map from
> SQLite's btree into Berkeley DB's. It looks like the patch is used not
> used.
> > http://src.chromium.org/cgi-
> bin/gitweb.cgi?p=chromium.git&a=search&h=HEAD&st=grep&s=sqlite3Poison
> >
> > safe-tolower.patch is very straight forward, it changes three lines
> of code. It looks like it can easily be applied by hand.
> >
> > preload-cache.patch can be replaced by an empty function:
> > int sqlite3Preload(sqlite3 *db) { return SQLITE_OK; }
> > and a prototype in sqlite.h.in
> > int sqlite3Preload(sqlite3 *);
> >
> > misc.patch has a bunch of small configuration settings and other
> minor changes, all easily integrated some already in our tree. You can
> skip removing the ifdef around sqlite3PcacheGetCachesize() because it's
> not used in our (empty) version of the sqlite3Preload() call (above).
> >
> >
> >
> > Liam, are these instructions enough for you to give it a try or do
> you need more help from us?
> >
> > -greg
> >
> >
> >
> >> -----Original Message-----
> >> From: Sandra Whitman
> >> Sent: Wednesday, March 03, 2010 9:51 AM
> >> To: Greg Burd
> >> Cc: bdb-beta-feedback at oss.oracle.com
> >> Subject: Re: [BDB 11gR2 Beta] Using FTS2 with the BDB SQLite
> >> compatibility library code [#18385]
> >>
> >> Hi Greg,
> >>
> >>
> >> I applied the 6 Chromium SQLite patches to our 5.0.11 source this
> >> morning. I did get rejects/filures for:
> >> misc.patch
> >> preload-cache.patch
> >> safe-tolower.patch
> >> sqlite-poison.patch
> >>
> >>
> >> I will try a manual fts2 build on top of this, but I'll have to get
> the
> >> patch failures straightened out.
> >>
> >>
> >> Thanks,
> >> Sandra
> >>
> >> Hexxeh wrote:
> >>
> >>> Hi,
> >>>
> >>> The patches that are applied to SQLite are as follows:
> >>>
> >>> http://src.chromium.org/cgi-
> >>>
> >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/sqlite-
> >> poison.patch;h=38f81a0c52fa2ff59b0e536f2418b4e003f97644;hb=HEAD
> >>
> >>> http://src.chromium.org/cgi-
> >>>
> >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/preload-
> >> cache.patch;h=8dc5778a03a29aa14d3416c83da61db3ccfdfa7c;hb=HEAD
> >>
> >>> http://src.chromium.org/cgi-
> >>>
> >>
> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/fts2.patch;h=
> >> 1f02161ee29aba22edad93ad7bba7c244e2dc2a1;hb=HEAD
> >>
> >>> http://src.chromium.org/cgi-
> >>>
> >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/icu-
> >> regexp.patch;h=cb758ce062a3dc6a0112ace3cdd136c18c7f8274;hb=HEAD
> >>
> >>> http://src.chromium.org/cgi-
> >>>
> >>
> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/misc.patch;h=
> >> e2f0c5525053a958bdd59019ab89d028ec201a97;hb=HEAD
> >>
> >>> http://src.chromium.org/cgi-
> >>>
> >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/safe-
> >> tolower.patch;h=c42bb065b7302ad1897be7c50dab59a3eb10c54e;hb=HEAD
> >>
> >>> You can see notes on what they changed and from which version these
> >>> patches are made
> >>> here: http://src.chromium.org/cgi-
> >>>
> >>
> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromi
> >> um;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD
> >>
> >>> Thanks,
> >>> Liam
> >>>
> >>> On 2 March 2010 22:44, Sandra Whitman <sandra.whitman at oracle.com
> >>> <mailto:sandra.whitman at oracle.com>> wrote:
> >>>
> >>> Hi Liam,
> >>>
> >>> Can you send us the patches/source code changes made for the
> >>> Chromium web browser, along with any particular directions for
> >>> installing them? I think we need to apply these as a starting
> >>>
> >> point.
> >>
> >>> Thanks,
> >>> Sandra
> >>>
> >>>
> >>> Hexxeh wrote:
> >>>
> >>> Yep, that's right. There's an option (just like with
> Firefox)
> >>> to use the system version of SQLite which after a little of
> >>>
> >> of
> >>
> >>> fiddling works as you'd expect.
> >>>
> >>> Liam
> >>>
> >>> On 2 March 2010 20:54, Greg Burd <GREG.BURD at oracle.com
> >>> <mailto:GREG.BURD at oracle.com> <mailto:GREG.BURD at oracle.com
> >>> <mailto:GREG.BURD at oracle.com>>> wrote:
> >>>
> >>> Sandra,
> >>>
> >>> Chromium OS is a Linux variant built by Google
> (sometimes
> >>> called
> >>> the "Google OS") which boots up and then uses the Google
> >>> browser,
> >>> Chromium (aka Chrome), for everything. Essentially,
> this
> >>>
> >> is
> >>
> >>> almost identical to what we've done with Firefox
> porting.
> >>> Liam is
> >>> replacing the Chromium web browser's use of SQLite with
> >>>
> >> BDB
> >>
> >>> SQL.
> >>> That then runs on this stripped down version of Linux
> and
> >>> runs on
> >>> a 2GB USB.
> >>>
> >>> When Liam looked into the way that Chromium web browser
> >>>
> >> used
> >>
> >>> SQLite he found that it:
> >>> a) modified it, patching bugs, adding external functions
> >>> using the
> >>> SQLite extensions API
> >>> b) used FTS2 (not FTS3) for full-text search
> >>> c) linked this statically against the browser
> >>>
> >>> For us to help Liam we will need to help him:
> >>> a) determine what patches to apply and how to apply them
> >>>
> >> to BDB
> >>
> >>> SQL (if necessary)
> >>> b) integrate the Google custom functions
> >>> c) integrate FTS2 for full-text search
> >>>
> >>> I believe he's already found a way to link Chromium
> >>>
> >> against a
> >>
> >>> dynamic sqlite3.so library rather than statically
> >>> compile/link the
> >>> amalgamated SQLite source as it does today.
> >>>
> >>> Did I get that right Liam?
> >>>
> >>> -greg
> >>>
> >>> > -----Original Message-----
> >>> > From: Sandra Whitman
> >>> > Sent: Tuesday, March 02, 2010 3:04 PM
> >>> > To: Hexxeh
> >>> > Cc: bdb-beta-feedback at oss.oracle.com
> >>> <mailto:bdb-beta-feedback at oss.oracle.com>
> >>> <mailto:bdb-beta-feedback at oss.oracle.com
> >>> <mailto:bdb-beta-feedback at oss.oracle.com>>
> >>> > Subject: Re: [BDB 11gR2 Beta] Using FTS2 with the BDB
> >>>
> >> SQLite
> >>
> >>> > compatibility library code
> >>> >
> >>> > Hi Liam,
> >>> >
> >>> > I am not familiar with the Chromium OS, but if it is
> >>> UNIX-like
> >>> then any
> >>> > arguments which can be passed to a standard SQLite
> >>>
> >> configure
> >>
> >>> script can
> >>> > also be passed to the Berkeley DB configure script.
> For
> >>> example, I
> >>> > typically build:
> >>> >
> >>> > ../dist/configure CPPFLAGS="-DSQLITE_ENABLE_FTS3"
> >>> --enable-sql
> >>> > or
> >>> > ../dist/configure CPPFLAGS="-DSQLITE_ENABLE_FTS3"
> >>> --enable-sql_compat
> >>> >
> >>> > As Dave mentioned, let me know the problem you are
> >>>
> >> having
> >>
> >>> and I can
> >>> > check further.
> >>> >
> >>> > Thanks,
> >>> > Sandra
> >>> >
> >>> > Hexxeh wrote:
> >>> > > Hi all,
> >>> > >
> >>> > > I'm trying to integrate the new BDB SQLite
> compatible
> >>> library into
> >>> > > Chromium/ChromiumOS, and I'm encountering a few
> issues
> >>> that I hope
> >>> > one
> >>> > > of you may have the answer to.
> >>> > >
> >>> > > One requirement of Chromium is the FTS2 extension,
> has
> >>> anyone
> >>> managed
> >>> > > to successfully use this with the library? If so,
> how
> >>> did you
> >>> > > accomplish this?
> >>> > >
> >>> > > Thanks,
> >>> > > Liam McLoughlin
> >>> > >
> >>>
> >>> ----------------------------------------------------------
> --
> >>>
> >> ---------
> >>
> >>> > ---
> >>> > >
> >>> > > _______________________________________________
> >>> > > BDB-BETA-FEEDBACK mailing list
> >>> > > BDB-BETA-FEEDBACK at oss.oracle.com
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>>
> >>>
> >>> > > http://oss.oracle.com/mailman/listinfo/bdb-beta-
> >>>
> >> feedback
> >>
> >>> >
> >>> > _______________________________________________
> >>> > BDB-BETA-FEEDBACK mailing list
> >>> > BDB-BETA-FEEDBACK at oss.oracle.com
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>>
> >>>
> >>> > http://oss.oracle.com/mailman/listinfo/bdb-beta-
> feedback
> >>>
> >>>
> >>> -----------------------------------------------------------
> --
> >>>
> >> -----------
> >>
> >>> _______________________________________________
> >>> BDB-BETA-FEEDBACK mailing list
> >>> BDB-BETA-FEEDBACK at oss.oracle.com
> >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>
> >>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
> >>>
> >>>
> >>> -------------------------------------------------------------------
> --
> >>>
> >> ---
> >>
> >>> _______________________________________________
> >>> BDB-BETA-FEEDBACK mailing list
> >>> BDB-BETA-FEEDBACK at oss.oracle.com
> >>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
> >>>
More information about the BDB-BETA-FEEDBACK
mailing list