[BDB 11gR2 Beta] Using FTS2 with the BDB SQLite compatibility library code [#18385]
Hexxeh
hexxeh at hexxeh.net
Wed Mar 3 08:07:42 PST 2010
Hi,
Looks good, I'm having a go using those changes now, I'll let you know how
it goes.
I think the other problem encountered in the build was these functions that
haven't been exposed:
- Exposed three functions that deal with unused file descriptors in
152</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l152>
os_unix.c, to allow Chromium's Posix VFS implementation in
153</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l153>
WebKit/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
154</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l154>
to correctly implement the "unused file descriptors" logic in the
155</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l155>
xDlOpen() method. The new functions are
156</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l156>
chromium_sqlite3_get_reusable_file_handle(),
157</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l157>
chromium_sqlite3_update_reusable_file_handle() and
158</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l158>
chromium_sqlite3_destroy_reusable_file_handle(). Also, added the
159</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l159>
chromium_sqlite3_fill_in_unix_sqlite3_file() function that calls
160</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l160>
fillInUnixFile(), which will be made static again as soon as a
161</cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l161>
WebKit patch using the new function lands.
Liam
On 3 March 2010 15:53, Greg Burd <GREG.BURD at oracle.com> wrote:
> Interesting... I wonder if the "design flaw" applies when BDB is VACUUM'ed
> as well... maybe we're immune to this issue.
>
> -greg
>
> > -----Original Message-----
> > From: Sandra Whitman
> > Sent: Wednesday, March 03, 2010 10:35 AM
> > To: Greg Burd
> > Cc: bdb-beta-feedback at oss.oracle.com; Hexxeh
> > Subject: Re: [BDB 11gR2 Beta] Using FTS2 with the BDB SQLite
> > compatibility library code [#18385]
> >
> > Thanks,
> >
> >
> > I have made some of those changes, but not all. I will do that. Right
> > now I am trying to add in the fts2 manual build. I'll let you know how
> > that goes.
> >
> >
> > One thing Emily told me last evening is that although the SQLite
> > documentation version 3.5 is not compatible with fts2, from:
> > http://www.mail-archive.com/sqlite-users@sqlite.org/msg31627.html,
> > fts2 has a design flaw that would cause corruptions of fts2 index when
> > running VACUUM against db. However if the VACUUM is not run, it should
> > be fine.
> >
> >
> > Thanks,
> > Sandra
> >
> >
> > Greg Burd wrote:
> > > Sandra,
> > >
> > > sqlite-poison.patch fails because it changes the btree.c code which
> > is not a shared part of the BDB version. We use that file to map from
> > SQLite's btree into Berkeley DB's. It looks like the patch is used not
> > used.
> > > http://src.chromium.org/cgi-
> > bin/gitweb.cgi?p=chromium.git&a=search&h=HEAD&st=grep&s=sqlite3Poison
> > >
> > > safe-tolower.patch is very straight forward, it changes three lines
> > of code. It looks like it can easily be applied by hand.
> > >
> > > preload-cache.patch can be replaced by an empty function:
> > > int sqlite3Preload(sqlite3 *db) { return SQLITE_OK; }
> > > and a prototype in sqlite.h.in
> > > int sqlite3Preload(sqlite3 *);
> > >
> > > misc.patch has a bunch of small configuration settings and other
> > minor changes, all easily integrated some already in our tree. You can
> > skip removing the ifdef around sqlite3PcacheGetCachesize() because it's
> > not used in our (empty) version of the sqlite3Preload() call (above).
> > >
> > >
> > >
> > > Liam, are these instructions enough for you to give it a try or do
> > you need more help from us?
> > >
> > > -greg
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: Sandra Whitman
> > >> Sent: Wednesday, March 03, 2010 9:51 AM
> > >> To: Greg Burd
> > >> Cc: bdb-beta-feedback at oss.oracle.com
> > >> Subject: Re: [BDB 11gR2 Beta] Using FTS2 with the BDB SQLite
> > >> compatibility library code [#18385]
> > >>
> > >> Hi Greg,
> > >>
> > >>
> > >> I applied the 6 Chromium SQLite patches to our 5.0.11 source this
> > >> morning. I did get rejects/filures for:
> > >> misc.patch
> > >> preload-cache.patch
> > >> safe-tolower.patch
> > >> sqlite-poison.patch
> > >>
> > >>
> > >> I will try a manual fts2 build on top of this, but I'll have to get
> > the
> > >> patch failures straightened out.
> > >>
> > >>
> > >> Thanks,
> > >> Sandra
> > >>
> > >> Hexxeh wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> The patches that are applied to SQLite are as follows:
> > >>>
> > >>> http://src.chromium.org/cgi-
> > >>>
> > >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/sqlite-
> > >> poison.patch;h=38f81a0c52fa2ff59b0e536f2418b4e003f97644;hb=HEAD
> > >>
> > >>> http://src.chromium.org/cgi-
> > >>>
> > >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/preload-
> > >> cache.patch;h=8dc5778a03a29aa14d3416c83da61db3ccfdfa7c;hb=HEAD
> > >>
> > >>> http://src.chromium.org/cgi-
> > >>>
> > >>
> > bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/fts2.patch;h=
> > >> 1f02161ee29aba22edad93ad7bba7c244e2dc2a1;hb=HEAD
> > >>
> > >>> http://src.chromium.org/cgi-
> > >>>
> > >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/icu-
> > >> regexp.patch;h=cb758ce062a3dc6a0112ace3cdd136c18c7f8274;hb=HEAD
> > >>
> > >>> http://src.chromium.org/cgi-
> > >>>
> > >>
> > bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/misc.patch;h=
> > >> e2f0c5525053a958bdd59019ab89d028ec201a97;hb=HEAD
> > >>
> > >>> http://src.chromium.org/cgi-
> > >>>
> > >> bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/safe-
> > >> tolower.patch;h=c42bb065b7302ad1897be7c50dab59a3eb10c54e;hb=HEAD
> > >>
> > >>> You can see notes on what they changed and from which version these
> > >>> patches are made
> > >>> here: http://src.chromium.org/cgi-
> > >>>
> > >>
> > bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromi
> > >> um;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD
> > >>
> > >>> Thanks,
> > >>> Liam
> > >>>
> > >>> On 2 March 2010 22:44, Sandra Whitman <sandra.whitman at oracle.com
> > >>> <mailto:sandra.whitman at oracle.com>> wrote:
> > >>>
> > >>> Hi Liam,
> > >>>
> > >>> Can you send us the patches/source code changes made for the
> > >>> Chromium web browser, along with any particular directions for
> > >>> installing them? I think we need to apply these as a starting
> > >>>
> > >> point.
> > >>
> > >>> Thanks,
> > >>> Sandra
> > >>>
> > >>>
> > >>> Hexxeh wrote:
> > >>>
> > >>> Yep, that's right. There's an option (just like with
> > Firefox)
> > >>> to use the system version of SQLite which after a little of
> > >>>
> > >> of
> > >>
> > >>> fiddling works as you'd expect.
> > >>>
> > >>> Liam
> > >>>
> > >>> On 2 March 2010 20:54, Greg Burd <GREG.BURD at oracle.com
> > >>> <mailto:GREG.BURD at oracle.com> <mailto:GREG.BURD at oracle.com
> > >>> <mailto:GREG.BURD at oracle.com>>> wrote:
> > >>>
> > >>> Sandra,
> > >>>
> > >>> Chromium OS is a Linux variant built by Google
> > (sometimes
> > >>> called
> > >>> the "Google OS") which boots up and then uses the Google
> > >>> browser,
> > >>> Chromium (aka Chrome), for everything. Essentially,
> > this
> > >>>
> > >> is
> > >>
> > >>> almost identical to what we've done with Firefox
> > porting.
> > >>> Liam is
> > >>> replacing the Chromium web browser's use of SQLite with
> > >>>
> > >> BDB
> > >>
> > >>> SQL.
> > >>> That then runs on this stripped down version of Linux
> > and
> > >>> runs on
> > >>> a 2GB USB.
> > >>>
> > >>> When Liam looked into the way that Chromium web browser
> > >>>
> > >> used
> > >>
> > >>> SQLite he found that it:
> > >>> a) modified it, patching bugs, adding external functions
> > >>> using the
> > >>> SQLite extensions API
> > >>> b) used FTS2 (not FTS3) for full-text search
> > >>> c) linked this statically against the browser
> > >>>
> > >>> For us to help Liam we will need to help him:
> > >>> a) determine what patches to apply and how to apply them
> > >>>
> > >> to BDB
> > >>
> > >>> SQL (if necessary)
> > >>> b) integrate the Google custom functions
> > >>> c) integrate FTS2 for full-text search
> > >>>
> > >>> I believe he's already found a way to link Chromium
> > >>>
> > >> against a
> > >>
> > >>> dynamic sqlite3.so library rather than statically
> > >>> compile/link the
> > >>> amalgamated SQLite source as it does today.
> > >>>
> > >>> Did I get that right Liam?
> > >>>
> > >>> -greg
> > >>>
> > >>> > -----Original Message-----
> > >>> > From: Sandra Whitman
> > >>> > Sent: Tuesday, March 02, 2010 3:04 PM
> > >>> > To: Hexxeh
> > >>> > Cc: bdb-beta-feedback at oss.oracle.com
> > >>> <mailto:bdb-beta-feedback at oss.oracle.com>
> > >>> <mailto:bdb-beta-feedback at oss.oracle.com
> > >>> <mailto:bdb-beta-feedback at oss.oracle.com>>
> > >>> > Subject: Re: [BDB 11gR2 Beta] Using FTS2 with the BDB
> > >>>
> > >> SQLite
> > >>
> > >>> > compatibility library code
> > >>> >
> > >>> > Hi Liam,
> > >>> >
> > >>> > I am not familiar with the Chromium OS, but if it is
> > >>> UNIX-like
> > >>> then any
> > >>> > arguments which can be passed to a standard SQLite
> > >>>
> > >> configure
> > >>
> > >>> script can
> > >>> > also be passed to the Berkeley DB configure script.
> > For
> > >>> example, I
> > >>> > typically build:
> > >>> >
> > >>> > ../dist/configure CPPFLAGS="-DSQLITE_ENABLE_FTS3"
> > >>> --enable-sql
> > >>> > or
> > >>> > ../dist/configure CPPFLAGS="-DSQLITE_ENABLE_FTS3"
> > >>> --enable-sql_compat
> > >>> >
> > >>> > As Dave mentioned, let me know the problem you are
> > >>>
> > >> having
> > >>
> > >>> and I can
> > >>> > check further.
> > >>> >
> > >>> > Thanks,
> > >>> > Sandra
> > >>> >
> > >>> > Hexxeh wrote:
> > >>> > > Hi all,
> > >>> > >
> > >>> > > I'm trying to integrate the new BDB SQLite
> > compatible
> > >>> library into
> > >>> > > Chromium/ChromiumOS, and I'm encountering a few
> > issues
> > >>> that I hope
> > >>> > one
> > >>> > > of you may have the answer to.
> > >>> > >
> > >>> > > One requirement of Chromium is the FTS2 extension,
> > has
> > >>> anyone
> > >>> managed
> > >>> > > to successfully use this with the library? If so,
> > how
> > >>> did you
> > >>> > > accomplish this?
> > >>> > >
> > >>> > > Thanks,
> > >>> > > Liam McLoughlin
> > >>> > >
> > >>>
> > >>> ----------------------------------------------------------
> > --
> > >>>
> > >> ---------
> > >>
> > >>> > ---
> > >>> > >
> > >>> > > _______________________________________________
> > >>> > > BDB-BETA-FEEDBACK mailing list
> > >>> > > BDB-BETA-FEEDBACK at oss.oracle.com
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>>
> > >>>
> > >>> > > http://oss.oracle.com/mailman/listinfo/bdb-beta-
> > >>>
> > >> feedback
> > >>
> > >>> >
> > >>> > _______________________________________________
> > >>> > BDB-BETA-FEEDBACK mailing list
> > >>> > BDB-BETA-FEEDBACK at oss.oracle.com
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>>
> > >>>
> > >>> > http://oss.oracle.com/mailman/listinfo/bdb-beta-
> > feedback
> > >>>
> > >>>
> > >>> -----------------------------------------------------------
> > --
> > >>>
> > >> -----------
> > >>
> > >>> _______________________________________________
> > >>> BDB-BETA-FEEDBACK mailing list
> > >>> BDB-BETA-FEEDBACK at oss.oracle.com
> > >>> <mailto:BDB-BETA-FEEDBACK at oss.oracle.com>
> > >>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
> > >>>
> > >>>
> > >>> -------------------------------------------------------------------
> > --
> > >>>
> > >> ---
> > >>
> > >>> _______________________________________________
> > >>> BDB-BETA-FEEDBACK mailing list
> > >>> BDB-BETA-FEEDBACK at oss.oracle.com
> > >>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
> > >>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100303/90453171/attachment-0001.html
More information about the BDB-BETA-FEEDBACK
mailing list