[BDB 11gR2 Beta] Using FTS2 with the BDB SQLite compatibility library code [#18385]

[Sandra Whitman]

Hello,


As of yesterday, Liam was still unable to build the fts2 support.  After 
speaking with Alex and Emily, it looks like there are a couple of ways 
to do this.  There are quite a few issues either way.  I need some 
guidance at this point to determine which way is really the right way to 
do this. 


1. My first try was basically to edit db-5.0.11/sql/sqlite/Makefile.in 
to build fts2 and then make in build_unix/sql.   I had to add the BDB 
library built in build_unix to the link line for libsqlite3.so to 
resolve a host of BDB symbols.

This results in a sqlite3 which at least has preliminary fts2 support:

db-5.0.11/build_unix/sql
[swhitman at swhitman-lap sql]$ ./sqlite3
Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February 19, 2010)
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
dbsql> CREATE VIRTUAL TABLE fortune using FTS2(class, text);
dbsql> insert into fortune (class, text) values ('class1', 'text1');
dbsql> insert into fortune (class, text) values ('class2', 'text2');
dbsql> select rowid, class, text from fortune where class match 'class1';
1|class1|text1
dbsql>


Alex pointed out that this method is not correct because the 
build_unix/sql build is not using the amalgamation, where as the build 
in build_unix is.  Hence there is a conflict and so the above is not the 
proper build method.


2. Based on Alex's input I tried:

$ cd dist
$ ./s_sql
(say yes to any permission related questions)
$ cd ../build_unix
$ ../dist/configure --enable-sql_compat
$ make

But here we've incorporated no fts2 support:
./sqlite3
Berkeley DB 11g Release 2, library version 11.2.5.0.11: (February 19, 2010)
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
dbsql>  CREATE VIRTUAL TABLE fortune using FTS2(class, text);
Error: no such module: FTS2


This might be the way to go, but how do I get fts2 in there?


3. Based on discussion with Emily I tried building with
/sql/sqlite/tool/mksqlite3c.tcl  editing this for fts2 support.


There are like a billion errors building this way.  I've cleaned up a 
lot, like 3/4 of a billion, but some still remain, and I'm not sure this 
is the way to go.  The cleanup for fts2 support involves some simple and 
some less simple changes:


1. editing the makefile to remove fts1 and fts3 support, otherwise there 
are conflicting functions with the same names, and adding include paths 
for db.h

2. configuring with:
../configure 
CFLAGS=-DSQLITE_ENABLE_BROKEN_FTS2,-DSQLITE_ENABLE_FTS2,-DSQLITE_MUTEX_PTHREADS

3. changing src/main.c, src/util.c, src/hash.c to ifdef to support fts2 
and ifdef out functions that conflict with the fts2 definitions.

4. I still get various combinations of SQLITE_PRIVATE/SQLITE_API/static 
errors that seem to need to be manually edited from sqlite3.c to avoid 
conflicts.  There are also some casts that need to be added for fts2 
functionality.

There are still errors regarding errno.h and dbinc/errno.h which I have 
not yet figured out. But I want to be sure this is a reasonable  
approach before I spend any more time.  I don't think it is.


At this point, is there any of the above or something else that seems 
like the best way to incorporate fts2?  I will work on whatever that 
choice is, but I am not sure at the moment which that is.


Thanks very much,
Sandra

Alex Gorrod wrote:
> Hi,
>
> On 4/03/2010 3:07 AM, Hexxeh wrote:
>> I think the other problem encountered in the build was these functions
>> that haven't been exposed:
>>
>>   - Exposed three functions that deal with unused file descriptors in
>> 152
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l152> 
>>
>>     os_unix.c, to allow Chromium's Posix VFS implementation in
>> 153
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l153> 
>>
>>     
>> WebKit/WebCore/platform/sql/chromium/SQLiteFileSystemChromiumPosix.cpp
>> 154
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l154> 
>>
>>     to correctly implement the "unused file descriptors" logic in the
>> 155
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l155> 
>>
>>     xDlOpen() method. The new functions are
>> 156
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l156> 
>>
>>     chromium_sqlite3_get_reusable_file_handle(),
>> 157
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l157> 
>>
>>     chromium_sqlite3_update_reusable_file_handle() and
>> 158
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l158> 
>>
>>     chromium_sqlite3_destroy_reusable_file_handle(). Also, added the
>> 159
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l159> 
>>
>>     chromium_sqlite3_fill_in_unix_sqlite3_file() function that calls
>> 160
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l160> 
>>
>>     fillInUnixFile(), which will be made static again as soon as a
>> 161
>> </cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=third_party/sqlite/README.chromium;h=6953e64e7bf0c462c10d0c7b84ff0052638f8e46;hb=HEAD#l161> 
>>
>>     WebKit patch using the new function lands.
>
> Could you clarify what this patch is used for please. I'll add some 
> context. Berkeley DB SQL does not use the SQLite VFS code - it is one 
> of the components that has been replaced.
>
> So, if those functions need to be exposed so that you can modify the 
> behavior of SQLite (by implementing a custom VFS), then your changes 
> should no longer be necessary.
>
> Thanks,
> Alex