[BDB 11gR2 Beta] Using FTS2 with the BDB SQLite compatibility library code [#18385]
Sandra Whitman
sandra.whitman at oracle.com
Fri Mar 5 19:39:54 PST 2010
Hi Emily,
I forgot that I did edit the Makefile.in sql/sqlite/Makefile.in so that
the fts3/fts1 functions are removed. I also added the include path to
pick up the db.h. You can just use this and forget about the generated
Makefile.
Thanks,
Sandra
Sandra Whitman wrote:
> Hi Emily,
>
>
> If we are going to try to build fts2 using sqlite/tool/mksqlite3c.tcl,
> here is what I've done. The files I changed are attached. If this is
> the way to go, I'll keep working on resolving the remaining
> compilation errors.
>
>
> 0. ../configure
> CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS
>
>
>
> 1. edit sql/sqlite/tool/mksqlite3c.tcl to remove fts1/fts3 support to
> avoid conflicting functions, and add fts2 in.
>
>
> 2. edit Makefile with includes to pickup db.h, otherwise the BDB
> symbols are undefined.
> TCC = gcc -DSQLITE_OS_UNIX=1 -I. -I${TOP}/src -I${TOP}/../adapter
> TCC += -I${TOP}/../../build_unix -I${TOP}/../../dist
>
>
> 3. edit in sql/sqlite/src/main.c, hash.c, util.c to add fts2 support
> into main.c, and remove routines in hash.c and util.c that conflict
> with fts2 versions. In hash.c and util.c I added #ifndef
> SQLITE_ENABLE_FTS2 around the routines that conflict with fts2 versions
>
> 4. There are some errors around the routines below. Mostly they are
> fixed by editing sqlite3.c to change SQLITE_PRIVATE/static to
> SQLITE_API. Otherwise I needed to add a cast as follows to
> findElementGivenHash:
> (fts2HashElem *)findElementGivenHash
>
> sqlite3.c:87016: error: conflicting types for ‘sqlite3PutVarint’
> sqlite3.c:10075: error: previous declaration of ‘sqlite3PutVarint’ was
> here
> sqlite3.c:87049: error: conflicting types for ‘sqlite3GetVarint’
> sqlite3.c:10077: error: previous declaration of ‘sqlite3GetVarint’ was
> here
> sqlite3.c:93965: error: static declaration of ‘strHash’ follows
> non-static declaration
> sqlite3.c:19467: error: previous implicit declaration of ‘strHash’ was
> here
> sqlite3.c:94038: warning: conflicting types for ‘insertElement’
> sqlite3.c:94038: error: static declaration of ‘insertElement’ follows
> non-static declaration
> sqlite3.c:19528: error: previous implicit declaration of
> ‘insertElement’ was here
> sqlite3.c:94062: warning: conflicting types for ‘rehash’
> sqlite3.c:94062: error: static declaration of ‘rehash’ follows
> non-static declaration
> sqlite3.c:19522: error: previous implicit declaration of ‘rehash’ was
> here
> sqlite3.c:94090: error: conflicting types for ‘findElementGivenHash’
> sqlite3.c:19471: error: previous implicit declaration of
> ‘findElementGivenHash’ was here
> sqlite3.c:94117: warning: conflicting types for ‘removeElementGivenHash’
> sqlite3.c:94117: error: static declaration of ‘removeElementGivenHash’
> follows non-static declaration
> sqlite3.c:19506: error: previous implicit declaration of
> ‘removeElementGivenHash’ was
>
> These should be ultimately fixed in the source, but for now I edited
> sqlite3.c.
>
>
> 5. After all of this there are a few remaining compilation errors. A
> few I can not figure out yet. One looks to be redefinition problem
> between errno.h and the dbinc/errno.h.
>
>
> Thanks,
> Sandra
>
> Sandra Whitman wrote:
>> Hello,
>>
>>
>> As of yesterday, Liam was still unable to build the fts2 support.
>> After speaking with Alex and Emily, it looks like there are a couple
>> of ways to do this. There are quite a few issues either way. I need
>> some guidance at this point to determine which way is really the
>> right way to do this.
>>
>> 1. My first try was basically to edit
>> db-5.0.11/sql/sqlite/Makefile.in to build fts2 and then make in
>> build_unix/sql. I had to add the BDB library built in build_unix to
>> the link line for libsqlite3.so to resolve a host of BDB symbols.
>>
>> This results in a sqlite3 which at least has preliminary fts2 support:
>>
>> db-5.0.11/build_unix/sql
>> [swhitman at swhitman-lap sql]$ ./sqlite3
>> Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February 19,
>> 2010)
>> Enter ".help" for instructions
>> Enter SQL statements terminated with a ";"
>> dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
>> dbsql> insert into fortune (class, text) values ('class1', 'text1');
>> dbsql> insert into fortune (class, text) values ('class2', 'text2');
>> dbsql> select rowid, class, text from fortune where class match
>> 'class1';
>> 1|class1|text1
>> dbsql>
>>
>>
>> Alex pointed out that this method is not correct because the
>> build_unix/sql build is not using the amalgamation, where as the
>> build in build_unix is. Hence there is a conflict and so the above
>> is not the proper build method.
>>
>>
>> 2. Based on Alex's input I tried:
>>
>> $ cd dist
>> $ ./s_sql
>> (say yes to any permission related questions)
>> $ cd ../build_unix
>> $ ../dist/configure --enable-sql_compat
>> $ make
>>
>> But here we've incorporated no fts2 support:
>> ./sqlite3
>> Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February 19,
>> 2010)
>> Enter ".help" for instructions
>> Enter SQL statements terminated with a ";"
>> dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
>> Error: no such module: FTS2
>>
>>
>> This might be the way to go, but how do I get fts2 in there?
>>
>>
>> 3. Based on discussion with Emily I tried building with
>> /sql/sqlite/tool/mksqlite3c.tcl editing this for fts2 support.
>>
>>
>> There are like a billion errors building this way. I've cleaned up a
>> lot, like 3/4 of a billion, but some still remain, and I'm not sure
>> this is the way to go. The cleanup for fts2 support involves some
>> simple and some less simple changes:
>>
>>
>> 1. editing the makefile to remove fts1 and fts3 support, otherwise
>> there are conflicting functions with the same names, and adding
>> include paths for db.h
>>
>> 2. configuring with:
>> ../configure
>> CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS
>>
>>
>> 3. changing src/main.c, src/util.c, src/hash.c to ifdef to support
>> fts2 and ifdef out functions that conflict with the fts2 definitions.
>>
>> 4. I still get various combinations of
>> SQLITE_PRIVATE/SQLITE_API/static errors that seem to need to be
>> manually edited from sqlite3.c to avoid conflicts. There are also
>> some casts that need to be added for fts2 functionality.
>>
>> There are still errors regarding errno.h and dbinc/errno.h which I
>> have not yet figured out. But I want to be sure this is a reasonable
>> approach before I spend any more time. I don't think it is.
>>
>>
>> At this point, is there any of the above or something else that seems
>> like the best way to incorporate fts2? I will work on whatever that
>> choice is, but I am not sure at the moment which that is.
>>
>>
>> Thanks very much,
>> Sandra
>>
>> Alex Gorrod wrote:
>>
>>> Hi,
>>>
>>> On 4/03/2010 3:07 AM, Hexxeh wrote:
>>>
>>>> I think the other problem encountered in the build was these functions
>>>> that haven't been exposed:
>>>>
>>>> - Exposed three functions that deal with unused file descriptors in
>>>> 152
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l152>
>>>>
>>>> os_unix.c, to allow Chromium's Posix VFS implementation in
>>>> 153
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l153>
>>>>
>>>>
>>>> WebKit/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>>>> 154
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l154>
>>>>
>>>> to correctly implement the "unused file descriptors" logic in the
>>>> 155
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l155>
>>>>
>>>> xDlOpen() method. The new functions are
>>>> 156
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l156>
>>>>
>>>> chromium_sqlite3_get_reusable_file_handle(),
>>>> 157
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l157>
>>>>
>>>> chromium_sqlite3_update_reusable_file_handle() and
>>>> 158
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l158>
>>>>
>>>> chromium_sqlite3_destroy_reusable_file_handle(). Also, added the
>>>> 159
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l159>
>>>>
>>>> chromium_sqlite3_fill_in_unix_sqlite3_file() function that calls
>>>> 160
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l160>
>>>>
>>>> fillInUnixFile(), which will be made static again as soon as a
>>>> 161
>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l161>
>>>>
>>>> WebKit patch using the new function lands.
>>>>
>>> Could you clarify what this patch is used for please. I'll add some
>>> context. Berkeley DB SQL does not use the SQLite VFS code - it is
>>> one of the components that has been replaced.
>>>
>>> So, if those functions need to be exposed so that you can modify the
>>> behavior of SQLite (by implementing a custom VFS), then your changes
>>> should no longer be necessary.
>>>
>>> Thanks,
>>> Alex
>>>
>>
>> _______________________________________________
>> BDB-BETA-FEEDBACK mailing list
>> BDB-BETA-FEEDBACK at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
>>
> ------------------------------------------------------------------------
>
> _______________________________________________
> BDB-BETA-FEEDBACK mailing list
> BDB-BETA-FEEDBACK at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile.in
Url: http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/5a69ecbf/attachment-0001.pl
More information about the BDB-BETA-FEEDBACK
mailing list