Subscribe: MySQL Forums - Custom Storage Engines
http://forums.mysql.com/feed.php?94,type=rss
Added By: Feedage Forager Feedage Grade B rated
Language: English
Tags:
bin mysqld  cursor  data  engine  handler unbind  handler  mysql  mysqld  product  replies  storage engine  storage  unbind psiev 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: MySQL Forums - Custom Storage Engines

MySQL Forums - Custom Storage Engines



Forum for your own Custom Storage Engines.



Last Build Date: Tue, 20 Feb 2018 10:19:19 +0000

 



Optimising select count(*) (no replies)

Thu, 23 Mar 2017 08:08:26 +0000

Hi here,

When a user tries to get count of rows in the table, mysql performs FULLSCAN by calling rnd_*() functions.

I'm sure there must be a way to say optimiser that I can extract this information from index. Could you please advise me how I can work it out?



List of requested fields (columns) from a table (no replies)

Thu, 08 Dec 2016 22:24:04 +0000

hello ppl, I'm writing a columnar storage engine and have a simple version working. however, during table scans or index scans I need to know what fields (columns) were requested by the user. given that this is a columnar db, I only want to fetch the fields the user wants.

can someone pls help me with this question?

thanks
Nishant



an error occured when build a custom storage engine (no replies)

Sun, 09 Oct 2016 10:35:50 +0000

Hi!

I want to expand the function of example storage engine.Then i write an extra source code files to define my function such as "indexdef.h" "index.c",and they were used in ha_example.cc and included in ha_example.h. After that i modified the CMakeList.txt to "SET(EXAMPLE_SOURCES index.c ha_fastidx.cc)".but when i build them ,it's occured to be failure,the function defined in my file can't be find:
...
...
[ 72%] Built target base64-t
[ 72%] Built target bitmap-t
[ 72%] Built target gen_lex_hash
[ 72%] Built target GenServerSource
[ 72%] Built target partition
[ 94%] Built target sql
Linking CXX executable explain_filename-t
../../storage/example/libexample.a(ha_example.cc.o): In function `ha_example::create(char const*, TABLE*, st_ha_create_information*)
/home/dbuser/share/sun/mysql-5.5.46/storage/example/ha_example.cc:1102: undefined reference to `aligned_malloc(long)'
/home/dbuser/share/sun/mysql-5.5.46/storage/example/ha_example.cc:1109: undefined reference to `isBigEndian()'
../../storage/example/libexample.a(ha_example.cc.o): In function `ha_example::write_row(unsigned char*)':
/home/dbuser/share/sun/mysql-5.5.46/storage/example/ha_example.cc:460: undefined reference to `atomic_fast_insert(st_fast_keydef*, u
collect2: ld returned 1 exit status
make[2]: *** [unittest/mysys/explain_filename-t] Error 1
make[1]: *** [unittest/mysys/CMakeFiles/explain_filename-t.dir/all] Error 2
make: *** [all] Error 2

how can i solve the problem ?



Index number mapping (2 replies)

Sun, 18 Sep 2016 23:38:02 +0000

Reading the documentation here:

https://dev.mysql.com/doc/internals/en/implementing-records-in-range-method.html

I see that the 'records_in_range' method (and others) take an uint to refer to the index to use for the method, but I don't understand when/where this mapping from integers to indices is defined.

For instance, when a table is created with a primary key and an index on the primary key, what is the resulting number of that index?

I also don't see anywhere in the documentation about how indices actually get created. Perhaps this is where the storage engine allocates ids for the created indices?

Could someone please point me in the right direction here?

Thanks,
Jonathan



Creating Custom Distributed Storage Engine (1 reply)

Sun, 18 Sep 2016 23:35:08 +0000

Hi All,

I saw the documentation here:

https://dev.mysql.com/doc/internals/en/custom-engine.html

But I wasn't sure if this documentation was valid for the case where I want to create a distributed, shared, storage backend for MySQL, where many MySQL servers are all accessing the same storage backend.

Could someone please point me in the right direction? I'm not sure if there's something special that goes into creating a distributed storage engine plugin for MySQL (say, for Cassandra, where one intends to deploy multiple MySQL servers that all access the same Cassandra cluster).

Thanks for any help,
Jonathan



Use field->store() in custom storage engine (1 reply)

Sat, 30 Jan 2016 19:46:32 +0000

Hello everyone

Does anybody know how the field->store() method works in custom storage engines?

I'd like to use it in the rnd_next(uchar *buf) method to save random data. I tried the following code:

char *data = (char*)malloc(field->max_display_length() +1);
*data = 's';
int data_len = sizeof(data) / sizeof(*char);
field->set_notnull();
field->store(data, data_len, &my_charset_bin);
free(data)

There were no compilingerrors, but the server crashes when i try to select some data of a table with my custom storage engine type.

Looking foreward for some answers



Custom server output (1 reply)

Mon, 15 Feb 2016 01:46:07 +0000

Hello everyone

I'd like to print out some messages like InnoDB does, when my mysqlserver starts.

I use the following command to launch the server:
sudo /path/to/server/mysqld --debug

I'd like to get an output like this:
2016-01-25T09:40:40.00000000 0 [Note] CustomEngine: Engine ready

Looking foreward for some answers



Fix links in Welcome post (no replies)

Thu, 14 Jan 2016 23:13:53 +0000

Can you guys fix the links in the Welcome post? I tried to follow them and they both return 404.

Thanks



Multiple full table scan during filesort (no replies)

Fri, 08 Jan 2016 19:40:52 +0000

Hello!

I'm creating custom storage engine for integration with our FullText retrieval DB.

Suppose that I have this table:

CREATE TABLE `test`.`bar` (
`rowid` int,
`ndoc` int,
`id` varchar(10) NOT NULL default '.',
`random` VARCHAR(50) NOT NULL,
`dictionary` VARCHAR(256),
`text` longtext,
`xml` longtext,
`extra` text,
PRIMARY KEY (`rowid`),
INDEX `ix_ndoc` (`ndoc`),
INDEX `ix_id` (`id`),
INDEX `ix_random` (`random`)
)
ENGINE = XWEE
DEFAULT CHARACTER SET = utf8;

When I execute this query:


select rowid, ndoc, id, random, dictionary, `text`
from test.bar
where (rowid is not null or extra in ('[xml,/bar/dictionary]=PRECO*'))
order by id ;

Consider that Where condition is always true cause its wanted that MySql dont check it,
I use pushed down condition to filter data.

My Problem is:

My Sql do correctly 2 times table scan, cause first time it should load columns necessary for file sort, then it should do file sort
and then it should do second table scan using sorted data, It work as is, BUT

It ask for all columns in both scans, AND that's the problem calculation for text column is very expensive for me
and it is not necessary for sort, I thought that in read_set it should flag just wanted columns, but it apparently does not function
cause it ask in bot runs all used columns.

What I'm doing wrong?

Thank you

ladislav



embedded server cannot accept a plugin engine (1 reply)

Fri, 13 Mar 2015 02:44:06 +0000

Hi,

I have developed a storage engine, ha_mybase.so, which works just fine with MySQL 5.6.23 used as a server. However, when I tried to load the engine into my application program using the embedded server, libmysqld.a, the loading is always rejected as shown below:

my_app_using_mysql> INSTALL PLUGIN fpcbase SONAME 'ha_mybase.so';
ERROR 1126: Can't open shared library '/usr/local/mysql/lib/plugin/ha_mybase.so' (errno: 2 /usr/local/mysql/lib/plugin/ha_mybase.so: undefined symbol: _ZN7handler10unbind_psiEv)

Note that the function symbol, _ZN7handler10unbind_psiEv, is indeed available in libmysqld.a as shown at the end of this post, but the application program keeps rejecting the loading. My application program is compiled using:

linux> g++ -I/usr/loca/mysql/include -L/usr/local/mysql/lib myapp.c -o myapp -lmysqld -lpthread -lm -lrt -lcrypt -ldl -laio -fno-rttiG -fno-rtti

Many attempts have been made to load the plugin engine with no luck, including
(1) using server_options
(2) using --mybase=FORCE_PLUS_PERMANENT in CMake and/or Makefile

Any help is greatly appreciated.

-- Steve

----------------------------------------------
linux> nm libmysqld.a |grep _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
U _ZN7handler10unbind_psiEv
0000000000000130 T _ZN7handler10unbind_psiEv



Storage Engine or Plugin (no replies)

Sat, 15 Nov 2014 15:46:49 +0000

Hi,

What is the difference between Storage Engine and Plugin generally (Not specific to Innodb)?

Thank you



Cloud Storage Engine for MySQL is generally available (no replies)

Tue, 15 Jul 2014 21:03:39 +0000

OblakSoft is pleased to announce general availability of the Cloud Storage Engine for MySQL (ClouSE) version 1.0. For more information visit http://www.oblaksoft.com/clouse-is-generally-available/.

-Artem



Comprehensive list of MySQL Storage Engines?? (no replies)

Thu, 06 Feb 2014 17:07:00 +0000

I would be most greatful if someone could point me to a comprehensive list of storage engines compatible with MySQL

Does anyone know of a comprehensive list of storage engines:
e.g
SPHINX
LUCENE
ARIA
:


And better still what storage engines provide both Transactional Processing like InnoDB as well as full text indexing and can be used in clustered environments?



MYSQL Key Partitioning - Uneven Distribution of data across partitions (no replies)

Fri, 30 Aug 2013 21:27:59 +0000

Hi

We are facing issues with uneven distribution of data across key partitions. The partition key is a UUID column. When we create 36 partitions, only 9 partitions are populated and when we create 100 partitions, only 25 partitions get populated. What could be the cause of that ? Is there some issue with MYSQL key partition while working with UUIDs?

Also, is there a way to customize the MYSQL key partitioning implementation so that we could use something like mod(crc32(),36)

Thanks for your help.
Prikshat



- (no replies)

Sat, 16 Mar 2013 01:16:17 +0000

test



Cursor fetching NULL values into variables (no replies)

Sat, 22 Dec 2012 06:43:54 +0000

Hi,

I am working on a solution in which I have to read data from a table parse it and load them into the target table. I though of starting with a sample code that will read data from the source table using cursor and display them. However, My cursor is fetching NULL values into the variables.
The source table has data. I tested the SELECT statement used in the cursor declaration seperately and it is working as expected.

Below is the steps I followed. Am I missing something? Is there something wrong with the code?

thanks in advance.


-- Creating the schema
CREATE SCHEMA `test_cursor` ;

-- Creating the table
CREATE TABLE `product` (
`product_id` int(5) NOT NULL AUTO_INCREMENT,
`product_code` varchar(5) DEFAULT NULL,
PRIMARY KEY (`product_id`)
) ENGINE=InnoDB ;

-- Inserting test data into the table
INSERT INTO `test_cursor`.`product` (`product_code`) VALUES ('prodA');
INSERT INTO `test_cursor`.`product` (`product_code`) VALUES ('prodB');
INSERT INTO `test_cursor`.`product` (`product_code`) VALUES ('prodC');
INSERT INTO `test_cursor`.`product` (`product_code`) VALUES ('prodD');
INSERT INTO `test_cursor`.`product` (`product_code`) VALUES ('prodE');

-- Procedure
DELIMITER $$
CREATE PROCEDURE `test_cursor`.`P_READ_PRODUCT` ()
BEGIN
/*
The procedure fetches data from the source product table and displays the same.
*/
-- VARIABLE/CONSTANT DECLARATION
DECLARE v_done INT DEFAULT FALSE; -- Variable used in the continue handler.
-- variables used to store the values fetched by the cursor itr_product from product table.
DECLARE a int(5);
DECLARE b varchar(5);

-- Declaration of cursor for iterating through the unprocessed records in the source table.
DECLARE itr_product CURSOR FOR SELECT product_id, product_code FROM test_cursor.product;

DECLARE CONTINUE HANDLER FOR NOT FOUND SET v_done=TRUE; -- Declaring continue handler for the cursor stg_product_data

OPEN itr_product; -- Open the cursor
read_loop: LOOP -- Loop through the records
FETCH itr_product INTO a, b; -- Fetching data into variables
IF v_done THEN -- Checking if the cursor has fetched the last record
LEAVE read_loop; -- Exit if last record had been fetched
ELSE
SELECT @a, @b ; -- Displaying the values fetched by the cursor
END IF;
END LOOP;
CLOSE itr_product; -- Close the cursor
END

-- Calling the procedure
call `test_cursor`.`P_READ_PRODUCT`();

-- Result set/Output
@a @b
null null



getting NULL value in uchar * buf in write_row(uchar *buf) (no replies)

Mon, 29 Oct 2012 06:01:06 +0000

Hi,

I am trying to write a custom storage engine. I have successfully implemented the create(), store_lock(), open() and rnd_init() functions following the guidelines from example storage engine. My implementation also more or less looks the same as in example storage engine. Now I wish to implement the write_row() function.
After successfully creating a table (through create table statement), I am trying to get uchar * buf from write_row(), but buf points to NULL everytime I try to debug this function. Also table->records[0] also gives NULL. Please help.

Thanks



internally used 3 or 4 byte (no replies)

Mon, 17 Sep 2012 14:09:50 +0000

We have a data store that is optimized for large scan and aggregation. Real simple example would be 'select Country, Sku, sum(Qty) from sales group by Country, Sku'. In addition to our storage format we have algorithms that use knowledge of that format to do the fast on demand aggregation. Right now the users interact with the data via a custom report type interface. We would like to add a SQL front end to the data store with out losing our aggregation strength. sons of anarchy season 5 episode 2

With out having looked into it, I am assuming that the interface between the sql engine and the storage engine is table based. That is the storage engine will return data to the sql engine as rows (I am sure this a simplification, the point being it is table oriented). watch the dark knight rises online The sql engine then does it's processing of the data (filtering, join, aggregating, etc...).

My question is does the api allow a 'push down' of the aggregation info so that the storage engine can return the data already aggregated? watch the expendables 2 online

if you mean DOS Kamenicky (aka keybcs2) or DOS Latin2 (aka cp852),
then MySQL does support both of them.
watch ted online
Otherwise, you can add your character set.
Starting from 4.1, it is to be done this way:

You need to edit:

/usr/local/mysql/share/charsets/exotic.xml
(by the way, what's its real name?)
watch horror movies online
Use say latin2.xml as an example.
You need to define these tables:
- Unicode mapping,
- lower and upper conversion
- ctype
- and sort order
basketball wives la season 2 episode 2



How to read and write table by MySQL 5.1 Storage Engine Plugin (no replies)

Wed, 12 Sep 2012 06:26:08 +0000

Hi All,

I want to create mysql storage plugin for read and write data into database table,

So how can i make this type of plugin and which header and lib files requaired for develope plugin, And if possible then can anyone give me smaple code for that into VC++ 2008.

Thank you all of you.

Thanks & Regards,

Ahir Vishal D.



Cursor Support in MySql (no replies)

Fri, 10 Aug 2012 10:36:42 +0000

Hi,

Currently in our application we have cursors through which we are calling procedures and now we are trying to migrate our application schema to MySQL Cluster. But we are unable migrate them as MySQL doesn't support procedure call with in the cluster.

Can somebody suggest if there is any alternative way to achive this.


Thanks,
Santosh



Need help: Facing issues with MySQL UDF implementations (no replies)

Fri, 10 Aug 2012 09:54:38 +0000

Hi I am trying to deply a MySQL cluster for in our telecome operating system. I could see MySQl supports only events which can be scheduled but I would like use events which can be posted by user as and when required. To implement this, I am trying to use MySQL UDF functions through which I can open a socket and send a message to server running in my application area. But I am facing some issues and when I call UDF function from mysql server it is getting shutdown and dumping below errors. Can somebody help me to resolve the issue? Thanks 09:10:29 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=2 max_threads=151 thread_count=2 connection_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337875 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x9538c10 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = ffffffffa7887390 thread_stack 0x30000 ../bin/mysqld(my_print_stacktrace+0x34)[0x8474785] ../bin/mysqld(handle_fatal_signal+0x2f9)[0x834d311] [0x110420] ../bin/mysqld(my_net_write+0xe7)[0x81d7d3b] ../bin/mysqld(_ZN8Protocol5writeEv+0x3e)[0x81dd100] ../bin/mysqld(_ZN11select_send9send_dataER4ListI4ItemE+0xd8)[0x821b1e2] ../bin/mysqld(_ZN4JOIN4execEv+0x336)[0x82602ba] ../bin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x2a9)[0x8261fad] ../bin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x151)[0x825c021] ../bin/mysqld[0x824460a] ../bin/mysqld(_Z21mysql_execute_commandP3THD+0x616)[0x823f2a4] ../bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1d7)[0x8245ce9] ../bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x7f9)[0x823d355] ../bin/mysqld(_Z10do_commandP3THD+0x11d)[0x823c9fb] ../bin/mysqld(_Z24do_handle_one_connectionP3THD+0x162)[0x82e3764] ../bin/mysqld(handle_one_connection+0x2b)[0x82e3319] /lib/libpthread.so.0[0x47b832] /lib/libc.so.6(clone+0x5e)[0x3bb46e] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (9501218): select myfunc_client("127.0.0.1") Connection ID (thread ID): 5 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Below is the UDF implementation. I am calling this function as below from MySQL. select myfunc_client("127.0.0.1"); int myfunc_client(UDF_INIT *initid __attribute__((unused)), UDF_ARGS *argv, char *is_null, char *error __attribute__((unused)),char* message __attribute__((unused))) { int sockfd = 0, n = 0 ,rc; char sendBuff[1024]; struct sockaddr_in serv_addr; char in_addr[256]; char msg[256] = "hello, sending message, to server from UDF&quo[...]



Retrieve original SQL statement from storage engine source code (1 reply)

Thu, 14 Jun 2012 13:31:31 +0000

Hello. I'm working on a read-only storage engine for academic purposes, and I can't figure out how to retrieve the original SQL statement (e.g. SELECT field_list FROM table) from the storage engine source code (more specifically, from inside the function rnd_init()). Is it possible?
I'm using MySQL version 5.6.5 on Debian Linux.
Thanks!



Using MySQL functions in Storage Engine (1 reply)

Thu, 29 Mar 2012 20:51:38 +0000

I'm generate a convention storeroom engine, and I would like to utilize some C++ functions, implemented in MySQL foundation code (for example: ha_rows filesort function from sql/filesort.cc)
What header files/libs must I include to achive this?



JSON Storage and Query Language Retrieval (no replies)

Mon, 16 Jan 2012 05:18:10 +0000

I have in-vain tried to find a pre-existing plugin, solution for dealing with json strings in mysql.

I would like to be able to store json packets in mysql. And be able to query them in standard SQL, in a map reduce approach.

I would have assumed something similar to what MSSQL provides in way of storing XML and then providing functions to query via xpath.

Hope my post is clear.

Regards,

The O



Range Queries on Custom Storage Engines (no replies)

Tue, 02 Aug 2011 02:25:34 +0000

Hello All,

I am just trying to figure out how a range query such as below mentioned one would pass on the storage engine.

select * from usertable where age < 45 and age > 40;

The custom engine only called for index search of 40 not for 45. I was assuming it would call for both values.

It seems that internally mysql engine is doing the filtering for 45.

Is that correct ? or am I missing something here ?

Thank You very much in advance.

- primal



_ZTV7handler not found on solaris (solved) (no replies)

Mon, 11 Jul 2011 22:04:00 +0000

Someone posted they had linking errors comping ha_example on Solaris. I've gotten this to work on Solaris, I even had the _ZTV7handler not found link error. I had to play around with the link flags.

Here are the relevant flags I am using. I compile with Solaris SunStudio CC. These flags may not be quite right if using GCC.

CXXFLAGS=-m64 -mt -KPIC -DPIC -G -g -DDEBUG -DMYSQL_DYNAMIC_PLUGIN -DHAVE_RWLOCK_T -DMYSQLDB -DSOLARIS -DUSE_HSREGEX

LDFLAGS=-m64 -G

I think it was the -KPIC and -DPIC that made the difference but I cant remember.



mysql cache getting in the way of my realtime custome engine (no replies)

Mon, 11 Jul 2011 21:43:55 +0000

I wrote a custom mysql engine that returns realtime data from one of our systems. The first time one of the "virtual" tables are queried I get the right results, but after that mysql caches this and it never changes. I am trying to figure out how to tell mysql optimizer that the data has changes and a rescan has to happen.

I figured it was part of the info() callback.

Can anyone point me in the right direction?

Thanks,
Colin



IBMDB2I and ON UPDATE CASCADE (no replies)

Fri, 13 May 2011 16:04:09 +0000

Hello,
I'm trying to create a table with a FK with "ON UPDATE CASCADE" using IBMDB2I storage engine on an IBM i (AS400). However apparently DB2 does not support ON UPDATE CASCADE and the create command fails. Has any one experienced this problem and/or has any ideas that may help? Thanks in advance.

-Hugo



Using MySQL functions in Storage Engine (no replies)

Wed, 02 Feb 2011 12:00:45 +0000

Hi!

I'm creating a custom storage engine, and I would like to use some C++ functions, implemented in MySQL source code (for example: ha_rows filesort function from sql/filesort.cc)

What header files/libs must I include to achive this?

Thanks,
Krisztian



Custom Storage Engines using Java (no replies)

Mon, 24 Jan 2011 18:30:53 +0000

Hi,

is there any possibility to write a custom storage engine in Java and load it into MySQL?

I could try compiling the java source using gcj to get the .so shared library for MySQL, I guess, however, this won't work because there is no way to include the MySQL header files required.

Thanks in advance,
Alex