DBF Append Speed in X# vs VO - Huge Difference

[Chris]

Hi Horst,

There have been a lot of improvements and fixes in the RDDs since 2020, and lately we haven't gotten any report about index corruption anymore, but of course something might have still slipped, a bug that we are not aware of. Did you make sure you had the index file properly opened when you edited the dbf from the tool?

[Chris]

Guys,

About dbf speed issues, there have been many improvements in the past months also in this area, but not all cases are improved. If you would like us to (try to) improve performance in your particular scenario, please send us a full sample showing the speed issue and we'll have a look.

[Horst]

Hello Chris

I was seeking with the tool, so i think the index was open.
If it will happend again, i try to reproduce the error and send you more informations.
On the files i am working with the x# app, i cant see troubles with the indexes.

Horst

[robert]

Horst,

If you're accessing the DBF/CDX with an external tool, make sure that the tool uses the same collation settings that your app uses.
Otherwise, corruption of the CDX can occur

Robert

[wriedmann]

Hi to all,
we have several projects were we are running both X# and VO applications on the same database, and we don't have registered index corruptions the last months.
About performance we cannot say a lot because in the network both VO and X# applications are running much slower than in the past, maybe because Microsoft make some changes in the network stack.
I will do some tests in the weekend.
Wolfgang
P.S. for the people that may interest that: all USB-C and Thunderbolt docking stations (both no name and large producers like HP, Dell and Lenovo) have really slow network interfaces, between 30 and 50% of the speed of either integrated network cards or dedicated USB-C network adapters

[Jamal]

Hi Chris,

For an example of the speed issue, please see the following post which is still relevant:

https://www.xsharp.eu/forum/topic?p=17140#p17140

Jamal

[Jamal]

Hey Guys,

I am still lurking around!

The speed issue is still in X# 2.21.

It took 22.51 minutes this time because I have a faster SSD on a new PC, but it is a long time. The dbf size after append is 20 MB.

The issue seems to occur when a DBFCDX order is active. When an order is not used, the loop is very fast and took only about 7 seconds.

Code:

Code: Select all

FUNCTION Start() AS VOID STRICT
    LOCAL cDBF AS STRING
    LOCAL aFields AS ARRAY
    LOCAL i AS DWORD

    LOCAL nStart, nElapsed AS REAL8

    RddSetDefault ( "DBFCDX" )

    cDBF := "C:\test\mytest"
    FErase ( cDBF + ".cdx" )

    aFields := { { "LAST" , "C" , 20 , 0 } }


    DbCreate( cDBF , aFields)
    DbUseArea( ,,cDBF )

    DBCreateOrder("last", "C:\test\mytest.cdx", "last")   // comment this out and it will be faster


   // wait

    nStart := Seconds()

    ? "start : " + ntrim(nStart)

    FOR i := 1 UPTO 1000000
        DbAppend()
        fieldput(1, asstring(i))
        // ? i
    NEXT

    nElapsed := Seconds() - nStart

    ? "Elapsed: " + ntrim(nElapsed) + " seconds" + " - Minutes: " + ntrim(nElapsed/60.00)

    wait


   Console.ReadKey()

Jamal

[Horst]

Hi Jamal

I tried it also with your sample. On my machine it took 50 min. !
And its the Index.
When i open the file shared and create the index at the end its 54s . In exclusive mode 19s

You can't make the index after the append ?

Horst

[Jamal]

Hi Horst,

Actually, that was a workaround which I am using now, but I wanted to mention that the issue still exists when an index file is active.

Hopefully, sometime in the future a fix is found.

Jamal

[Chris]

Guys,

The fundamental problem with this seems to be that in X# the time needed to complete the test grows exponentially with the number of records, while in VO it grows linearly. Will log this, maybe something can be changed to the index algorithm in X# to speed it up. Without an index, it grow linearly also in X#, which is good.

But is it a realistic real life scenario needing to add millions (or 100,000s) of records at once with the index activated?