Read performance of QSPI Flash

[wdx04]

Hi,

I did a little benchmark of read performance of the QSPI Flash on some of my STM32 boards.
The QSPI frequency passed to QSPIFBlockDevice's constructor is 100MHz.
I thought I could get at least 20-30MB/s speeds, but I only got these results:

Board	Flash Chip	Read Speed
STM32F746I-DISCO	N25Q128A	4.51MB/s
STM32F469I-DISCO	N25Q128A	2.47MB/s
Custom STM32H723ZG	W25Q64JV	0.51MB/s
Custom STM32U575VG	W25Q32JV	0.48MB/s

I know DMA is not used in current STM32 QSPI/OSPI implementation, so I won't get the speeds close to the theoretical maximum.
But these results are still quite disappointing, especially with the W25Qxx flash chips.
I even doubt I bought these boards with inferior W25Qxx chips by accident.
Has anyone gotten similar results to me? How much speed up can we expect if DMA is used?

Here is my code for testing the read performance of a block device:

void do_perf(BlockDevice *block_device)
{
    constexpr off_t read_buffer_size = 32768;
    static char read_file_buffer[read_buffer_size];
    if(block_device == nullptr)
    {
        printf("invalid block device.\n");
        return;
    }
    off_t total_size = std::min(off_t(8 * 1024 * 1024), (off_t) block_device->size());
    if(total_size <= 0)
    {
        // very large SD cards
        total_size = 8 * 1024 * 1024;
    }
    off_t source_size = total_size;
    bd_addr_t offset = 0;
    Timer perf_timer;
    perf_timer.start();
    while(source_size > 0)
    {
        off_t batch_size = source_size > read_buffer_size ? read_buffer_size: source_size;
        block_device->read(read_file_buffer, offset, batch_size);
        source_size -= batch_size;
        offset += batch_size;
    }
    perf_timer.stop();
    long long elapsed_time_ms = chrono::duration_cast<chrono::milliseconds>(perf_timer.elapsed_time()).count();
    printf("finished read %ld bytes in %lld ms.\n", total_size, elapsed_time_ms);
    if(elapsed_time_ms == 0)
    {
        printf("read time too short, errors occured!\n", elapsed_time_ms);
    }
    else
    {
        float speed_mbps = float(total_size) / float(1024 * 1024) / float(elapsed_time_ms) * 1000.0f;
        printf("read speed: %.2fMB/s\n", speed_mbps);
    }
}

I also tested the read performance of SD cards, with DMA and a SDIO clock overclocked to 200MHz, on a custom STM32H563VI board. The read performance is 57.14MB/s with a 32KB buffer and 86.02MB/s with a 320K(max size without SDRAM) buffer.

Thanks!

[multiplemonomials]

Huh, that is surprisingly bad... I would be interested to see a logic analyzer capture of running this benchmark. For instance:

• Is the flash actually in quad mode?
• Does the SCLK frequency match what's set?
• What percentage of the time is the serial clock active?

I feel like for the performance to be this bad, either Mbed has to be ignoring the clock settings, or there has to be something else other than transferring data from the flash that's taking up most of the time.

[wdx04]

Huh, that is surprisingly bad... I would be interested to see a logic analyzer capture of running this benchmark. For instance:

• Is the flash actually in quad mode?
• Does the SCLK frequency match what's set?
• What percentage of the time is the serial clock active?

I feel like for the performance to be this bad, either Mbed has to be ignoring the clock settings, or there has to be something else other than transferring data from the flash that's taking up most of the time.

Today I tested the read performance of raw QSPI class on a STM32L476 Discovery board, which also has a N25Q128 chip.
Besides, I also tested memory-mapped mode with a derived class:

class MappableQSPI : public QSPI
{
public:
    MappableQSPI(PinName io0, PinName io1, PinName io2, PinName io3, PinName sclk, PinName ssel = NC, int mode = 0)
        : QSPI(io0, io1, io2, io3, sclk, ssel, mode)
    {}

    MappableQSPI(const qspi_pinmap_t &pinmap, int mode = 0)
        : QSPI(pinmap, mode)
    {}

    bool map_memory()
    {
#ifdef QUADSPI
        // Map QSPI Memory in 1-4-4 mode
        QSPI_CommandTypeDef sCommand; 
        sCommand.DdrMode = QSPI_DDR_MODE_DISABLE; 
        sCommand.DdrHoldHalfCycle = QSPI_DDR_HHC_ANALOG_DELAY;
        sCommand.Instruction = CMD_READ;
        sCommand.InstructionMode = QSPI_INSTRUCTION_1_LINE;
        sCommand.AddressMode = QSPI_ADDRESS_4_LINES; 
        sCommand.AddressSize = QSPI_ADDRESS_24_BITS;
        sCommand.AlternateByteMode = QSPI_ALTERNATE_BYTES_NONE;
        sCommand.DataMode = QSPI_DATA_4_LINES;
        sCommand.DummyCycles = DUMMY_CYCLES;
        sCommand.SIOOMode = QSPI_SIOO_INST_EVERY_CMD;

        QSPI_MemoryMappedTypeDef sMemMappedCfg;
        sMemMappedCfg.TimeOutActivation = QSPI_TIMEOUT_COUNTER_DISABLE;
        sMemMappedCfg.TimeOutPeriod = 0;

        if (HAL_QSPI_MemoryMapped(&_qspi.handle, &sCommand, &sMemMappedCfg) != HAL_OK)
        {
            return false;
        }
        return true;
#else
        OSPI_MemoryMappedTypeDef sMemMappedCfg = {0};
        sMemMappedCfg.TimeOutActivation = HAL_OSPI_TIMEOUT_COUNTER_DISABLE;
        sMemMappedCfg.TimeOutPeriod = 0;
        if (HAL_OSPI_MemoryMapped(&_qspi.handle, &sMemMappedCfg) != HAL_OK)
        {
            return false;
        }
        return true;
#endif
    }
};

Here are the results (time for reading 8MB of data in 32KB blocks):

QSPI Freqency	QSPI Mode	QSPI::read Time(ms)	Memory Mapped+memcpy time(ms)
1MHz	1-1-1	67126	67108
10MHz	1-1-1	6911	6864
80MHz	1-1-1	6872	868
1MHz	1-4-4	16889	16895
10MHz	1-4-4	6873	1732
80MHz	1-4-4	6871(1.16MB/s)	217 (36.87MB/s)

As the results show, at very low frequencies, both increasing the frequency and switching to 4-line mode can significantly increase the read speed. But at 10MHz or higher, continuing to increase the frequency or switching to 4-line mode makes little differences. Conversely, in memory-mapped mode, we can always get the expected speedup from higher frequencies and 4-line mode.
This shows that the QSPI API overhead is very large.

[multiplemonomials]

It could also be that Mbed is simply deciding to not run in quad mode and not to use a frequency higher than 10MHz... I really think stepping through the QSPIF block device code or using a logic analyzer would be worth doing, if you're able. Thank you for looking in to this!

[wdx04]

It could also be that Mbed is simply deciding to not run in quad mode and not to use a frequency higher than 10MHz... I really think stepping through the QSPIF block device code or using a logic analyzer would be worth doing, if you're able. Thank you for looking in to this!

The read performance of raw QSPI class is bad and it does not depend on QSPIF block device code.
I've looked into the code of QSPI class and ST QSPI driver, there is no special treatment for specific frequeny ranges.
Also, it's unlikely that QSPI is not actually running at the specified frequency: HAL_QSPI_MemoryMapped is a single API call that's unrelated to the operating frequency, but after the HAL_QSPI_MemoryMapped call, I do get the read speed matching the specified frequency. I'll try to analyze the signals using a logic analyzer later, when I find a suitable board.

[wdx04]

Here are the logic analyzer captures on my STM32H723 board, under 10M, 20M and 40M frequencies:

Obviously, the QSPI(actually OSPI in QUAD mode) peripheral do work in 4-line mode, and the operating frequency is also right, but as the frequency increases, the interval after every 4 bytes received will increase, which makes the actual read speed not increase after reaching a certain frequency.

Note: QSPI::set_frequency does not work for OSPI peripherals in QUAD mode. They would be fixed at the initial ONE_MHZ frequency. That's why I got 0.5MB/s on STM32U575 and STM32H723 boards before. For testing purpose I modified the initial _hz value in QSPI's constructor.

[multiplemonomials]

Huh, interesting. So I guess there's just some delay in Mbed's QSPI or QSPIF code? Unfortunately I haven't worked with this area much, so you have probably more insight than me into what's going on here...

[wdx04]

I modified the ST QSPI driver to make use of DMA on L4, F4 and F7 MCUs. The speed boost ranges from 6.5x to 9.5x. (9.43MB/s on L476, 24.32MB/s on F469 and 29.09MB/s on F746). This is still significantly slower than memory-mapped mode. There might be something wrong.
STM32H7 it is a bit harder because QSPI/OSPI requests are only supported by MDMA, and MDMA is not supported by stm_dma_utils for now.

[wdx04]

I have implemented the QSPI+DMA driver on STM32U5 and got a read speed of 37.74MB/s at 80MHz OSPI clock, very close to the speed of memory-mapped mode. It seems the performance of QSPI+DMA combination highly depends on the hardware implementation.

To support QSPI+DMA on STM32H7 I need to add MDMA functionality in stm_dma_utils. I have two choices:

1. Change the signature of DMA util functions, such as stm_init_dma_link, replacing DMA_HandleTypeDef* with void *. Because we need to support DMA1/DMA2 and MDMA at the same time, but they have different handle types(DMA_HandleTypeDef and MDMA_HandleTypeDef). Also their stream/channel types are different(DMA_Stream_TypeDef and MDMA_Channel_TypeDef).
2. Add a new group of util functions, for example stm_init_mdma_link, specially for MDMA

[multiplemonomials]

Hmm, could we define the DMA handle typedef as a union of those three types? I'd accept a patch for that!

[wdx04]

Hmm, could we define the DMA handle typedef as a union of those three types? I'd accept a patch for that!

I think I could define a variant-like struct that contains a union of handle pointer and an ID field indicating the actual type of the handle pointer.