Performance of TRIM command on ext4 filesystem

PostMark test

Parametres of the Postmark test
set number 50000
set transactions 50000
set read 4096
set write 4096
set bias read 5
set bias create 5
set report terse
set size 1000 200000
set buffering true|false
I have run this particular test on ext4 filesystem with -o nodiscard and -o discard mount options 10x with each mount option, then I have computed average values and percentage differences between nodiscard and discard option.

Testing order

1. mkfs.ext4
2. mount -o (nodiscard|discard)
3. ./postmark
4. umount
5. repeat 1. - 4. ten times 

vendor3 device

                                   |   	      BUFFERING ENABLED*        |       BUFFERING DISABLED*          |
--------------------------------------------------------------------------------------------------------------
Type				   |NODISCARD    DISCARD      DIFF      |NODISCARD    DISCARD      DIFF      |
==============================================================================================================
Total_duration                     |377.80       598.80       58.50%    |370.20       594.60       60.62%    |
Duration_of_transactions           |297.20       521.60       75.50%    |285.10       515.90       80.95%    |
Transactions/s                     |168.30       95.86        -43.04%   |175.51       96.93        -44.77%   |
Files_created/s                    |197.92       124.86       -36.91%   |201.98       125.75       -37.74%   |
Creation_alone/s                   |773.10       785.16       1.56%     |772.00       784.03       1.56%     |
Creation_mixed_with_transaction/s  |83.34        47.47        -43.04%   |86.92        48.00        -44.78%   |
Read/s                             |84.06        47.88        -43.04%   |87.67        48.41        -44.78%   |
Append/s                           |84.23        47.98        -43.04%   |87.85        48.51        -44.78%   |
Deleted/s                          |197.92       124.86       -36.91%   |201.98       125.75       -37.74%   |
Deletion_alone/s                   |765.71       777.65       1.56%     |764.62       776.53       1.56%     |
Deletion_mixed_with_transaction/s  |84.95        48.39        -43.04%   |88.59        48.93        -44.77%   |
Read_B/s                           |7288249.95   4597787.15   -36.92%   |7437901.50   4630656.70   -37.74%   |
Write_B/s                          |22807978.80  14388396.10  -36.92%   |23276301.20  14491259.30  -37.74%   |
==============================================================================================================
[nodiscard with buffering] [discard with buffering] [nodiscard without buffering] [discard without buffering]

vendor1 device

                                   |   	      BUFFERING ENABLED*        |       BUFFERING DISABLED*          |
--------------------------------------------------------------------------------------------------------------
Type				   |NODISCARD    DISCARD      DIFF      |NODISCARD    DISCARD      DIFF      |
==============================================================================================================
Total_duration                     |182.40       212.10       16.28%    |180.30       209.10       15.97%    |
Duration_of_transactions           |120.40       148.80       23.59%    |117.60       146.70       24.74%    |
Transactions/s                     |415.30       336.06       -19.08%   |425.20       340.84       -19.84%   |
Files_created/s                    |409.95       352.54       -14.00%   |414.73       357.56       -13.78%   |
Creation_alone/s                   |970.40       963.67       -0.69%    |954.59       966.58       1.26%     |
Creation_mixed_with_transaction/s  |205.67       166.42       -19.08%   |210.57       168.79       -19.84%   |
Read/s                             |207.44       167.85       -19.09%   |212.38       170.24       -19.84%   |
Append/s                           |207.86       168.19       -19.08%   |212.81       170.59       -19.84%   |
Deleted/s                          |409.95       352.54       -14.00%   |414.73       357.56       -13.78%   |
Deletion_alone/s                   |961.13       954.45       -0.70%    |945.46       957.33       1.26%     |
Deletion_mixed_with_transaction/s  |209.63       169.63       -19.08%   |214.63       172.05       -19.84%   |
Read_B/s                           |15096194.90  12981826.50  -14.01%   |15272242.70  13166944.50  -13.79%   |
Write_B/s                          |47242300.80  40625558.00  -14.01%   |47793228.00  41204869.60  -13.79%   |
==============================================================================================================
[nodiscard with buffering] [discard with buffering] [nodiscard without buffering] [discard without buffering]

vendor2

                                   |   	      BUFFERING ENABLED*        |       BUFFERING DISABLED*          |
--------------------------------------------------------------------------------------------------------------
Type				   |NODISCARD    DISCARD      DIFF      |NODISCARD    DISCARD      DIFF      |
==============================================================================================================
Total_duration                     |220.20       2628.50      1093.69%  |218.40       2651.60      1114.10%  |
Duration_of_transactions           |157.90       2479.90      1470.55%  |155.80       2493.50      1500.45%  |
Transactions/s                     |316.68       20.16        -93.63%   |320.93       20.06        -93.75%   |
Files_created/s                    |339.52       28.45        -91.62%   |342.32       28.20        -91.76%   |
Creation_alone/s                   |969.86       981.15       1.16%     |971.11       971.11       -0.00%    |
Creation_mixed_with_transaction/s  |156.83       9.99         -93.63%   |158.93       9.93         -93.75%   |
Read/s                             |158.17       10.07        -93.63%   |160.30       10.02        -93.75%   |
Append/s                           |158.50       10.09        -93.63%   |160.63       10.04        -93.75%   |
Deleted/s                          |339.52       28.45        -91.62%   |342.32       28.20        -91.76%   |
Deletion_alone/s                   |960.59       971.77       1.16%     |961.83       961.83       -0.00%    |
Deletion_mixed_with_transaction/s  |159.86       10.18        -93.63%   |162.00       10.12        -93.75%   |
Read_B/s                           |12502731.80  1047732.08   -91.62%   |12605752.30  1038640.13   -91.76%   |
Write_B/s                          |39126271.20  3278791.40   -91.62%   |39448664.80  3250338.88   -91.76%   |
==============================================================================================================
[nodiscard with buffering] [discard with buffering] [nodiscard without buffering] [discard without buffering]

vendor4

                                   |          BUFFERING ENABLED         |       BUFFERING DISABLED           |
--------------------------------------------------------------------------------------------------------------
Type                               |NODISCARD    DISCARD      DIFF      |NODISCARD    DISCARD      DIFF      |
==============================================================================================================
Total_duration                     |327.50       333.50       1.83%     |327.20       330.00       0.86%     |
Duration_of_transactions           |239.40       243.60       1.75%     |238.20       241.30       1.30%     |
Transactions/s                     |209.30       205.34       -1.89%    |209.96       207.25       -1.29%    |
Files_created/s                    |228.47       224.20       -1.87%    |228.52       226.58       -0.85%    |
Creation_alone/s                   |632.22       620.72       -1.82%    |623.66       627.51       0.62%     |
Creation_mixed_with_transaction/s  |103.65       101.69       -1.89%    |103.97       102.64       -1.28%    |
Read/s                             |104.54       102.56       -1.89%    |104.87       103.52       -1.29%    |
Append/s                           |104.76       102.77       -1.90%    |105.08       103.73       -1.28%    |
Deleted/s                          |228.47       224.20       -1.87%    |228.52       226.58       -0.85%    |
Deletion_alone/s                   |626.18       614.78       -1.82%    |617.70       621.51       0.62%     |
Deletion_mixed_with_transaction/s  |105.65       103.65       -1.89%    |105.98       104.62       -1.28%    |
Read_B/s                           |8413387.05   8256152.15   -1.87%    |8415148.45   8343683.55   -0.85%    |
Write_B/s                          |26329003.20  25836949.00  -1.87%    |26334515.00  26110871.40  -0.85%    |
==============================================================================================================
[nodiscard with buffering] [discard with buffering] [nodiscard without buffering] [discard without buffering]

vendor5

                                   |          BUFFERING ENABLED         |       BUFFERING DISABLED           |
--------------------------------------------------------------------------------------------------------------
Type                               |NODISCARD    DISCARD      DIFF      |NODISCARD    DISCARD      DIFF      |
==============================================================================================================
Total_duration                     |177.10       181.00       2.20%     |172.60       177.50       2.84%     |
Duration_of_transactions           |123.10       127.80       3.82%     |119.10       124.30       4.37%     |
Transactions/s                     |406.22       391.26       -3.68%    |419.84       402.27       -4.18%    |
Files_created/s                    |422.15       413.06       -2.15%    |433.16       421.21       -2.76%    |
Creation_alone/s                   |1018.53      1026.96      0.83%     |1020.58      1031.04      1.02%     |
Creation_mixed_with_transaction/s  |201.17       193.77       -3.68%    |207.91       199.22       -4.18%    |
Read/s                             |202.89       195.43       -3.68%    |209.70       200.93       -4.18%    |
Append/s                           |203.31       195.83       -3.68%    |210.13       201.34       -4.18%    |
Deleted/s                          |422.15       413.06       -2.15%    |433.16       421.21       -2.76%    |
Deletion_alone/s                   |1008.79      1017.14      0.83%     |1010.82      1021.18      1.02%     |
Deletion_mixed_with_transaction/s  |205.05       197.50       -3.68%    |211.92       203.06       -4.18%    |
Read_B/s                           |15545372.04  15210638.77  -2.15%    |15950766.86  15510515.62  -2.76%    |
Write_B/s                          |48647963.38  47600443.14  -2.15%    |49916613.13  48538883.09  -2.76%    |
==============================================================================================================
[nodiscard with buffering] [discard with buffering] [nodiscard without buffering] [discard without buffering]

vendor6

                                   |          BUFFERING ENABLED         |       BUFFERING DISABLED           |
--------------------------------------------------------------------------------------------------------------
Type                               |NODISCARD    DISCARD      DIFF      |NODISCARD    DISCARD      DIFF      |
==============================================================================================================
Total_duration                     |144.80       143.40       -0.97%    |140.00       138.60       -1.00%    |
Duration_of_transactions           |95.40        94.00        -1.47%    |90.60        90.40        -0.22%    |
Transactions/s                     |524.21       532.01       1.49%     |551.89       553.22       0.24%     |
Files_created/s                    |516.34       521.37       0.97%     |534.02       539.45       1.02%     |
Creation_alone/s                   |1116.16      1131.31      1.36%     |1121.21      1141.65      1.82%     |
Creation_mixed_with_transaction/s  |259.60       263.46       1.49%     |273.31       273.96       0.24%     |
Read/s                             |261.83       265.73       1.49%     |275.66       276.32       0.24%     |
Append/s                           |262.37       266.27       1.49%     |276.22       276.89       0.24%     |
Deleted/s                          |516.34       521.37       0.97%     |534.02       539.45       1.02%     |
Deletion_alone/s                   |1105.49      1120.50      1.36%     |1110.49      1130.73      1.82%     |
Deletion_mixed_with_transaction/s  |264.61       268.55       1.49%     |278.58       279.25       0.24%     |
Read_B/s                           |19013824.01  19199189.31  0.97%     |19664863.96  19864982.58  1.02%     |
Write_B/s                          |59502198.56  60082284.04  0.97%     |61539574.58  62165829.34  1.02%     |
==============================================================================================================
[nodiscard with buffering] [discard with buffering] [nodiscard without buffering] [discard without buffering]

Note: This test has been done a little bit differently, because I figure out that even that this device handles TRIM commands quite well, the performance of the device goes a bit down afterwards, as the firmware apparently process the request.

This behavior has bigger impact the bigger the TRIM range is. So after mke2fs (and whole device TRIM) there is some period of time when device performance is lower than normal.

The time it needs to completely process TRIM on background depends not only on the size of TRIM range, but also on how much writes as been done to the device between TRIM's.

There is a graph of performance stabilization right after the whole device trim. It has been measured without any writes in between, sadly I no longer have this device in my possessions so I can not test it.



* - Buffering means that C library function like fopen, fread, fwrite are used instead of open, read, write. I have used the word buffering in the same way as it is used in the postmark test