[BDB 11gR2 Beta] Using FTS2 with the BDB SQLite compatibility library code [#18385]
Sandra Whitman
sandra.whitman at oracle.com
Fri Mar 5 19:57:57 PST 2010
Oops, sorry for so many emails, I fixed the Makefile.in so that there
are no more compilation errors regarding errno.h.
There are actually just a few compilation errors left. I'll work on
those tomorrow. Most are the SQLITE_PRIVATE/SQLITE_API issues.
Thanks,
Sandra
Sandra Whitman wrote:
> Hi Emily,
>
> I forgot that I did edit the Makefile.in sql/sqlite/Makefile.in so
> that the fts3/fts1 functions are removed. I also added the include
> path to pick up the db.h. You can just use this and forget about the
> generated Makefile.
>
> Thanks,
> Sandra
>
> Sandra Whitman wrote:
>> Hi Emily,
>>
>>
>> If we are going to try to build fts2 using
>> sqlite/tool/mksqlite3c.tcl, here is what I've done. The files I
>> changed are attached. If this is the way to go, I'll keep working on
>> resolving the remaining compilation errors.
>>
>>
>> 0. ../configure
>> CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS
>>
>>
>>
>> 1. edit sql/sqlite/tool/mksqlite3c.tcl to remove fts1/fts3 support to
>> avoid conflicting functions, and add fts2 in.
>>
>>
>> 2. edit Makefile with includes to pickup db.h, otherwise the BDB
>> symbols are undefined.
>> TCC = gcc -DSQLITE_OS_UNIX=1 -I. -I${TOP}/src -I${TOP}/../adapter
>> TCC += -I${TOP}/../../build_unix -I${TOP}/../../dist
>>
>>
>> 3. edit in sql/sqlite/src/main.c, hash.c, util.c to add fts2 support
>> into main.c, and remove routines in hash.c and util.c that conflict
>> with fts2 versions. In hash.c and util.c I added #ifndef
>> SQLITE_ENABLE_FTS2 around the routines that conflict with fts2 versions
>>
>> 4. There are some errors around the routines below. Mostly they are
>> fixed by editing sqlite3.c to change SQLITE_PRIVATE/static to
>> SQLITE_API. Otherwise I needed to add a cast as follows to
>> findElementGivenHash:
>> (fts2HashElem *)findElementGivenHash
>>
>> sqlite3.c:87016: error: conflicting types for ‘sqlite3PutVarint’
>> sqlite3.c:10075: error: previous declaration of ‘sqlite3PutVarint’
>> was here
>> sqlite3.c:87049: error: conflicting types for ‘sqlite3GetVarint’
>> sqlite3.c:10077: error: previous declaration of ‘sqlite3GetVarint’
>> was here
>> sqlite3.c:93965: error: static declaration of ‘strHash’ follows
>> non-static declaration
>> sqlite3.c:19467: error: previous implicit declaration of ‘strHash’
>> was here
>> sqlite3.c:94038: warning: conflicting types for ‘insertElement’
>> sqlite3.c:94038: error: static declaration of ‘insertElement’ follows
>> non-static declaration
>> sqlite3.c:19528: error: previous implicit declaration of
>> ‘insertElement’ was here
>> sqlite3.c:94062: warning: conflicting types for ‘rehash’
>> sqlite3.c:94062: error: static declaration of ‘rehash’ follows
>> non-static declaration
>> sqlite3.c:19522: error: previous implicit declaration of ‘rehash’ was
>> here
>> sqlite3.c:94090: error: conflicting types for ‘findElementGivenHash’
>> sqlite3.c:19471: error: previous implicit declaration of
>> ‘findElementGivenHash’ was here
>> sqlite3.c:94117: warning: conflicting types for ‘removeElementGivenHash’
>> sqlite3.c:94117: error: static declaration of
>> ‘removeElementGivenHash’ follows non-static declaration
>> sqlite3.c:19506: error: previous implicit declaration of
>> ‘removeElementGivenHash’ was
>>
>> These should be ultimately fixed in the source, but for now I edited
>> sqlite3.c.
>>
>>
>> 5. After all of this there are a few remaining compilation errors. A
>> few I can not figure out yet. One looks to be redefinition problem
>> between errno.h and the dbinc/errno.h.
>>
>>
>> Thanks,
>> Sandra
>>
>> Sandra Whitman wrote:
>>> Hello,
>>>
>>>
>>> As of yesterday, Liam was still unable to build the fts2 support.
>>> After speaking with Alex and Emily, it looks like there are a couple
>>> of ways to do this. There are quite a few issues either way. I
>>> need some guidance at this point to determine which way is really
>>> the right way to do this.
>>>
>>> 1. My first try was basically to edit
>>> db-5.0.11/sql/sqlite/Makefile.in to build fts2 and then make in
>>> build_unix/sql. I had to add the BDB library built in build_unix
>>> to the link line for libsqlite3.so to resolve a host of BDB symbols.
>>>
>>> This results in a sqlite3 which at least has preliminary fts2 support:
>>>
>>> db-5.0.11/build_unix/sql
>>> [swhitman at swhitman-lap sql]$ ./sqlite3
>>> Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February
>>> 19, 2010)
>>> Enter ".help" for instructions
>>> Enter SQL statements terminated with a ";"
>>> dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
>>> dbsql> insert into fortune (class, text) values ('class1', 'text1');
>>> dbsql> insert into fortune (class, text) values ('class2', 'text2');
>>> dbsql> select rowid, class, text from fortune where class match
>>> 'class1';
>>> 1|class1|text1
>>> dbsql>
>>>
>>>
>>> Alex pointed out that this method is not correct because the
>>> build_unix/sql build is not using the amalgamation, where as the
>>> build in build_unix is. Hence there is a conflict and so the above
>>> is not the proper build method.
>>>
>>>
>>> 2. Based on Alex's input I tried:
>>>
>>> $ cd dist
>>> $ ./s_sql
>>> (say yes to any permission related questions)
>>> $ cd ../build_unix
>>> $ ../dist/configure --enable-sql_compat
>>> $ make
>>>
>>> But here we've incorporated no fts2 support:
>>> ./sqlite3
>>> Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February
>>> 19, 2010)
>>> Enter ".help" for instructions
>>> Enter SQL statements terminated with a ";"
>>> dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
>>> Error: no such module: FTS2
>>>
>>>
>>> This might be the way to go, but how do I get fts2 in there?
>>>
>>>
>>> 3. Based on discussion with Emily I tried building with
>>> /sql/sqlite/tool/mksqlite3c.tcl editing this for fts2 support.
>>>
>>>
>>> There are like a billion errors building this way. I've cleaned up
>>> a lot, like 3/4 of a billion, but some still remain, and I'm not
>>> sure this is the way to go. The cleanup for fts2 support involves
>>> some simple and some less simple changes:
>>>
>>>
>>> 1. editing the makefile to remove fts1 and fts3 support, otherwise
>>> there are conflicting functions with the same names, and adding
>>> include paths for db.h
>>>
>>> 2. configuring with:
>>> ../configure
>>> CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS
>>>
>>>
>>> 3. changing src/main.c, src/util.c, src/hash.c to ifdef to support
>>> fts2 and ifdef out functions that conflict with the fts2 definitions.
>>>
>>> 4. I still get various combinations of
>>> SQLITE_PRIVATE/SQLITE_API/static errors that seem to need to be
>>> manually edited from sqlite3.c to avoid conflicts. There are also
>>> some casts that need to be added for fts2 functionality.
>>>
>>> There are still errors regarding errno.h and dbinc/errno.h which I
>>> have not yet figured out. But I want to be sure this is a
>>> reasonable approach before I spend any more time. I don't think it
>>> is.
>>>
>>>
>>> At this point, is there any of the above or something else that
>>> seems like the best way to incorporate fts2? I will work on
>>> whatever that choice is, but I am not sure at the moment which that is.
>>>
>>>
>>> Thanks very much,
>>> Sandra
>>>
>>> Alex Gorrod wrote:
>>>
>>>> Hi,
>>>>
>>>> On 4/03/2010 3:07 AM, Hexxeh wrote:
>>>>
>>>>> I think the other problem encountered in the build was these
>>>>> functions
>>>>> that haven't been exposed:
>>>>>
>>>>> - Exposed three functions that deal with unused file descriptors in
>>>>> 152
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l152>
>>>>>
>>>>> os_unix.c, to allow Chromium's Posix VFS implementation in
>>>>> 153
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l153>
>>>>>
>>>>>
>>>>> WebKit/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>>>>>
>>>>> 154
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l154>
>>>>>
>>>>> to correctly implement the "unused file descriptors" logic in the
>>>>> 155
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l155>
>>>>>
>>>>> xDlOpen() method. The new functions are
>>>>> 156
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l156>
>>>>>
>>>>> chromium_sqlite3_get_reusable_file_handle(),
>>>>> 157
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l157>
>>>>>
>>>>> chromium_sqlite3_update_reusable_file_handle() and
>>>>> 158
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l158>
>>>>>
>>>>> chromium_sqlite3_destroy_reusable_file_handle(). Also, added the
>>>>> 159
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l159>
>>>>>
>>>>> chromium_sqlite3_fill_in_unix_sqlite3_file() function that calls
>>>>> 160
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l160>
>>>>>
>>>>> fillInUnixFile(), which will be made static again as soon as a
>>>>> 161
>>>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l161>
>>>>>
>>>>> WebKit patch using the new function lands.
>>>>>
>>>> Could you clarify what this patch is used for please. I'll add some
>>>> context. Berkeley DB SQL does not use the SQLite VFS code - it is
>>>> one of the components that has been replaced.
>>>>
>>>> So, if those functions need to be exposed so that you can modify
>>>> the behavior of SQLite (by implementing a custom VFS), then your
>>>> changes should no longer be necessary.
>>>>
>>>> Thanks,
>>>> Alex
>>>>
>>>
>>> _______________________________________________
>>> BDB-BETA-FEEDBACK mailing list
>>> BDB-BETA-FEEDBACK at oss.oracle.com
>>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
>>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> BDB-BETA-FEEDBACK mailing list
>> BDB-BETA-FEEDBACK at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
> ------------------------------------------------------------------------
>
> _______________________________________________
> BDB-BETA-FEEDBACK mailing list
> BDB-BETA-FEEDBACK at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile.in
Url: http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/8d5a832e/attachment-0001.pl
More information about the BDB-BETA-FEEDBACK
mailing list