5.2.10 Speed of INSERT Queries

The time to insert a record consists approximately of:

• Connect: (3)
• Sending query to server: (2)
• Parsing query: (2)
• Inserting record: (1 x size of record)
• Inserting indexes: (1 x number of indexes)
• Close: (1)

where the numbers are somewhat proportional to the overall time. This does not take into consideration the initial overhead to open tables (which is done once for each concurrently running query).

The size of the table slows down the insertion of indexes by log N (B-trees).

Some ways to speed up inserts:

• If you are inserting many rows from the same client at the same time, use multiple value lists INSERT statements. This is much faster (many times in some cases) than using separate INSERT statements. If you are adding data to non-empty table, you may tune up the bulk_insert_buffer_size variable to make it even faster. See section 4.5.7.4 SHOW VARIABLES.
• If you are inserting a lot of rows from different clients, you can get higher speed by using the INSERT DELAYED statement. See section 6.4.3 INSERT Syntax.
• Note that with MyISAM tables you can insert rows at the same time SELECTs are running if there are no deleted rows in the tables.
• When loading a table from a text file, use LOAD DATA INFILE. This is usually 20 times faster than using a lot of INSERT statements. See section 6.4.9 LOAD DATA INFILE Syntax.
• It is possible with some extra work to make LOAD DATA INFILE run even faster when the table has many indexes. Use the following procedure:
  1. Optionally create the table with CREATE TABLE. For example, using mysql or Perl-DBI.
  2. Execute a FLUSH TABLES statement or the shell command mysqladmin flush-tables.
  3. Use myisamchk --keys-used=0 -rq /path/to/db/tbl_name. This will remove all usage of all indexes from the table.
  4. Insert data into the table with LOAD DATA INFILE. This will not update any indexes and will therefore be very fast.
  5. If you are going to only read the table in the future, run myisampack on it to make it smaller. See section 7.1.2.3 Compressed Table Characteristics.
  6. Re-create the indexes with myisamchk -r -q /path/to/db/tbl_name. This will create the index tree in memory before writing it to disk, which is much faster because it avoids lots of disk seeks. The resulting index tree is also perfectly balanced.
  7. Execute a FLUSH TABLES statement or the shell command mysqladmin flush-tables.
  Note that LOAD DATA INFILE also does the above optimisation if you insert into an empty table; the main difference with the above procedure is that you can let myisamchk allocate much more temporary memory for the index creation that you may want MySQL to allocate for every index recreation. Since MySQL 4.0 you can also use ALTER TABLE tbl_name DISABLE KEYS instead of myisamchk --keys-used=0 -rq /path/to/db/tbl_name and ALTER TABLE tbl_name ENABLE KEYS instead of myisamchk -r -q /path/to/db/tbl_name. This way you can also skip FLUSH TABLES steps.
• You can speed up insertions that is done over multiple statements by locking your tables:

  mysql> LOCK TABLES a WRITE;
  mysql> INSERT INTO a VALUES (1,23),(2,34),(4,33);
  mysql> INSERT INTO a VALUES (8,26),(6,29);
  mysql> UNLOCK TABLES;
  

  The main speed difference is that the index buffer is flushed to disk only once, after all INSERT statements have completed. Normally there would be as many index buffer flushes as there are different INSERT statements. Locking is not needed if you can insert all rows with a single statement. For transactional tables, you should use BEGIN/COMMIT instead of LOCK TABLES to get a speedup. Locking will also lower the total time of multi-connection tests, but the maximum wait time for some threads will go up (because they wait for locks). For example:

  thread 1 does 1000 inserts
  thread 2, 3, and 4 does 1 insert
  thread 5 does 1000 inserts
  

  If you don't use locking, 2, 3, and 4 will finish before 1 and 5. If you use locking, 2, 3, and 4 probably will not finish before 1 or 5, but the total time should be about 40% faster. As INSERT, UPDATE, and DELETE operations are very fast in MySQL, you will obtain better overall performance by adding locks around everything that does more than about 5 inserts or updates in a row. If you do very many inserts in a row, you could do a LOCK TABLES followed by an UNLOCK TABLES once in a while (about each 1000 rows) to allow other threads access to the table. This would still result in a nice performance gain. Of course, LOAD DATA INFILE is much faster for loading data.

To get some more speed for both LOAD DATA INFILE and INSERT, enlarge the key buffer. See section 5.5.2 Tuning Server Parameters.

User Comments

There is a multi value example above. This one contains 3 rows to be inserterd.

mysql> INSERT INTO a VALUES (1,23),(2,34),(4,33);

The multivalue statement is not clear.

I assume the values correspond to the columns in the order they appear in the table?

What if the table has 9 columns but we only need to update 2 columns, and those 2 columns are in the 3rd and 7th positions?

What if we need to change the 3rd & 5th columns for one row, and the 6th & 9th columns for the another row?

Explicit usage, syntax and limitations would be helpful.

insert into table table_name (col5, col7) values (x1,y1), (x2,y2), (x3,y3)

mysql Ver 12.18 Distrib 4.0.12, for pc-linux (i686)


ALTER TABLE xxxx DISABLE KEYS;

Dosn't work - Indexes are still in force, and
are still updated as the table is.

Same goes for
myisamchk --keys-used=0 -rq /path/to/db/tbl_name
(same version). During updates, .MYI files are
updated too.

"If you are inserting a lot of rows from different clients, you can get higher speed by using the INSERT DELAYED statement."

I am running an application which inserts 20 to 120 rows per second on a single table (Format: Fixed/Row size: 46/No free data) using 5 to 10 client threads. I was using INSERT DELAYED but the delayed insert thread was consuming a lot of CPU and every 5 seconds all the insert threads would go into "waiting for handler lock" status for about 600 ms. The application also runs also selects and updates in the following proportion:
INSERT: 28%
SELECT: 28%
UPDATE: 44%

During that 600ms every 5 seconds, mysql didn't output anything to any thread (I checked it by monitoring network traffic).

Today I changed the INSERT DELAYED to a simple INSERT and CPU usage went down by 70% and no more intermitent locks are experienced.

Platform: Dual Intel Pentium III - Linux 2.4.20-13.7smp - MySQL 4.0.12-standard. Queries per second avg: 482.485

When needing to insert multiple value inserts with selected table columns, simply specify the columns as you do in a normal insert

i.e. with a table a with five columns a1, a2,a3,a4 and a5 you want to insert only to a3 and a5...

INSERT INTO a (a3,a5) VALUES (1,3) (3,6) (3,7);

When it comes to explicit syntax I would point to the manual which actually does indicate this and also to books.

Examples are few in the manual, but plenty on the net.

Add your own comment.