[BDB 11gR2 Beta] Using FTS2 with the BDB SQLite compatibility library code [#18385]
Sandra Whitman
sandra.whitman at oracle.com
Fri Mar 5 19:27:32 PST 2010
Hi Emily,
If we are going to try to build fts2 using sqlite/tool/mksqlite3c.tcl,
here is what I've done. The files I changed are attached. If this is the
way to go, I'll keep working on resolving the remaining compilation errors.
0. ../configure
CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS
1. edit sql/sqlite/tool/mksqlite3c.tcl to remove fts1/fts3 support to
avoid conflicting functions, and add fts2 in.
2. edit Makefile with includes to pickup db.h, otherwise the BDB symbols
are undefined.
TCC = gcc -DSQLITE_OS_UNIX=1 -I. -I${TOP}/src -I${TOP}/../adapter
TCC += -I${TOP}/../../build_unix -I${TOP}/../../dist
3. edit in sql/sqlite/src/main.c, hash.c, util.c to add fts2 support
into main.c, and remove routines in hash.c and util.c that conflict with
fts2 versions. In hash.c and util.c I added #ifndef SQLITE_ENABLE_FTS2
around the routines that conflict with fts2 versions
4. There are some errors around the routines below. Mostly they are
fixed by editing sqlite3.c to change SQLITE_PRIVATE/static to
SQLITE_API. Otherwise I needed to add a cast as follows to
findElementGivenHash:
(fts2HashElem *)findElementGivenHash
sqlite3.c:87016: error: conflicting types for ‘sqlite3PutVarint’
sqlite3.c:10075: error: previous declaration of ‘sqlite3PutVarint’ was here
sqlite3.c:87049: error: conflicting types for ‘sqlite3GetVarint’
sqlite3.c:10077: error: previous declaration of ‘sqlite3GetVarint’ was here
sqlite3.c:93965: error: static declaration of ‘strHash’ follows
non-static declaration
sqlite3.c:19467: error: previous implicit declaration of ‘strHash’ was here
sqlite3.c:94038: warning: conflicting types for ‘insertElement’
sqlite3.c:94038: error: static declaration of ‘insertElement’ follows
non-static declaration
sqlite3.c:19528: error: previous implicit declaration of ‘insertElement’
was here
sqlite3.c:94062: warning: conflicting types for ‘rehash’
sqlite3.c:94062: error: static declaration of ‘rehash’ follows
non-static declaration
sqlite3.c:19522: error: previous implicit declaration of ‘rehash’ was here
sqlite3.c:94090: error: conflicting types for ‘findElementGivenHash’
sqlite3.c:19471: error: previous implicit declaration of
‘findElementGivenHash’ was here
sqlite3.c:94117: warning: conflicting types for ‘removeElementGivenHash’
sqlite3.c:94117: error: static declaration of ‘removeElementGivenHash’
follows non-static declaration
sqlite3.c:19506: error: previous implicit declaration of
‘removeElementGivenHash’ was
These should be ultimately fixed in the source, but for now I edited
sqlite3.c.
5. After all of this there are a few remaining compilation errors. A few
I can not figure out yet. One looks to be redefinition problem between
errno.h and the dbinc/errno.h.
Thanks,
Sandra
Sandra Whitman wrote:
> Hello,
>
>
> As of yesterday, Liam was still unable to build the fts2 support. After
> speaking with Alex and Emily, it looks like there are a couple of ways
> to do this. There are quite a few issues either way. I need some
> guidance at this point to determine which way is really the right way to
> do this.
>
>
> 1. My first try was basically to edit db-5.0.11/sql/sqlite/Makefile.in
> to build fts2 and then make in build_unix/sql. I had to add the BDB
> library built in build_unix to the link line for libsqlite3.so to
> resolve a host of BDB symbols.
>
> This results in a sqlite3 which at least has preliminary fts2 support:
>
> db-5.0.11/build_unix/sql
> [swhitman at swhitman-lap sql]$ ./sqlite3
> Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February 19, 2010)
> Enter ".help" for instructions
> Enter SQL statements terminated with a ";"
> dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
> dbsql> insert into fortune (class, text) values ('class1', 'text1');
> dbsql> insert into fortune (class, text) values ('class2', 'text2');
> dbsql> select rowid, class, text from fortune where class match 'class1';
> 1|class1|text1
> dbsql>
>
>
> Alex pointed out that this method is not correct because the
> build_unix/sql build is not using the amalgamation, where as the build
> in build_unix is. Hence there is a conflict and so the above is not the
> proper build method.
>
>
> 2. Based on Alex's input I tried:
>
> $ cd dist
> $ ./s_sql
> (say yes to any permission related questions)
> $ cd ../build_unix
> $ ../dist/configure --enable-sql_compat
> $ make
>
> But here we've incorporated no fts2 support:
> ./sqlite3
> Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February 19, 2010)
> Enter ".help" for instructions
> Enter SQL statements terminated with a ";"
> dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
> Error: no such module: FTS2
>
>
> This might be the way to go, but how do I get fts2 in there?
>
>
> 3. Based on discussion with Emily I tried building with
> /sql/sqlite/tool/mksqlite3c.tcl editing this for fts2 support.
>
>
> There are like a billion errors building this way. I've cleaned up a
> lot, like 3/4 of a billion, but some still remain, and I'm not sure this
> is the way to go. The cleanup for fts2 support involves some simple and
> some less simple changes:
>
>
> 1. editing the makefile to remove fts1 and fts3 support, otherwise there
> are conflicting functions with the same names, and adding include paths
> for db.h
>
> 2. configuring with:
> ../configure
> CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS
>
> 3. changing src/main.c, src/util.c, src/hash.c to ifdef to support fts2
> and ifdef out functions that conflict with the fts2 definitions.
>
> 4. I still get various combinations of SQLITE_PRIVATE/SQLITE_API/static
> errors that seem to need to be manually edited from sqlite3.c to avoid
> conflicts. There are also some casts that need to be added for fts2
> functionality.
>
> There are still errors regarding errno.h and dbinc/errno.h which I have
> not yet figured out. But I want to be sure this is a reasonable
> approach before I spend any more time. I don't think it is.
>
>
> At this point, is there any of the above or something else that seems
> like the best way to incorporate fts2? I will work on whatever that
> choice is, but I am not sure at the moment which that is.
>
>
> Thanks very much,
> Sandra
>
> Alex Gorrod wrote:
>
>> Hi,
>>
>> On 4/03/2010 3:07 AM, Hexxeh wrote:
>>
>>> I think the other problem encountered in the build was these functions
>>> that haven't been exposed:
>>>
>>> - Exposed three functions that deal with unused file descriptors in
>>> 152
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l152>
>>>
>>> os_unix.c, to allow Chromium's Posix VFS implementation in
>>> 153
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l153>
>>>
>>>
>>> WebKit/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>>> 154
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l154>
>>>
>>> to correctly implement the "unused file descriptors" logic in the
>>> 155
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l155>
>>>
>>> xDlOpen() method. The new functions are
>>> 156
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l156>
>>>
>>> chromium_sqlite3_get_reusable_file_handle(),
>>> 157
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l157>
>>>
>>> chromium_sqlite3_update_reusable_file_handle() and
>>> 158
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l158>
>>>
>>> chromium_sqlite3_destroy_reusable_file_handle(). Also, added the
>>> 159
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l159>
>>>
>>> chromium_sqlite3_fill_in_unix_sqlite3_file() function that calls
>>> 160
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l160>
>>>
>>> fillInUnixFile(), which will be made static again as soon as a
>>> 161
>>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l161>
>>>
>>> WebKit patch using the new function lands.
>>>
>> Could you clarify what this patch is used for please. I'll add some
>> context. Berkeley DB SQL does not use the SQLite VFS code - it is one
>> of the components that has been replaced.
>>
>> So, if those functions need to be exposed so that you can modify the
>> behavior of SQLite (by implementing a custom VFS), then your changes
>> should no longer be necessary.
>>
>> Thanks,
>> Alex
>>
>
> _______________________________________________
> BDB-BETA-FEEDBACK mailing list
> BDB-BETA-FEEDBACK at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/bdb-beta-feedback
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hash.c
Type: text/x-csrc
Size: 7891 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/a34bbc72/attachment-0004.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.c
Type: text/x-csrc
Size: 69151 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/a34bbc72/attachment-0005.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: util.c
Type: text/x-csrc
Size: 28981 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/a34bbc72/attachment-0006.bin
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Makefile
Url: http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/a34bbc72/attachment-0001.pl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mksqlite3c.tcl
Type: text/x-tcl
Size: 7911 bytes
Desc: not available
Url : http://oss.oracle.com/pipermail/bdb-beta-feedback/attachments/20100305/a34bbc72/attachment-0007.bin
More information about the BDB-BETA-FEEDBACK
mailing list