Re: [Dar-libdar_api] Poor performance of unlimited integer size

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 10/08/2018 12:00, Dennis Katsonis wrote:
> On 08/10/2018 06:29 AM, Denis Corbin wrote:
>> On 09/08/2018 13:37, Dennis Katsonis wrote:
>>> Hello,
> 
>> Hello Dennis,
> 
> 
>>> I am developing a front end for Dar which is intended not just
>>> to provide a graphical way of creating archive, but also
>>> provide basic backup management.  The application will be
>>> written using the Qt toolkit and using libdar directly.
> 
>> nice! :)
> 
> 
> I am still in two minds about it.  It is intended to be 
> backup-focused, for those who like manual, simple backups, but
> with more emphasis on making restoration and viewing and managing
> the state of the backups easy and visible.  I'm not sure whether
> this should be the separate project I've started, or a contribution
> to KDar which includes dar_database style management
> functionality.

I can't tell you what's the best direction here, but I just remember
since Johnathan K. Burchill developed kdar (started in 2003) that
there was some issues with the new KDE version some years later (I
guess it was KDE 4)... issues that have not been solved AFAIK (correct
me if I'm wrong).

> 
> I should give my thanks to you for creating dar, as I was looking
> for a replacement for dump/restore and it met all my needs.
> Simple, easy differential backup, able to do ad-hoc backups,
> backups in manageable file archives, encryption and can reliably
> save ALL the files attributes easily.  It is a software package
> where you can tell the author has given a lot of thought to how it
> might be used and what people might want to do, and accommodated
> for that and documented it wel l.

Thanks!

> 

[...]

> 
>> the memory requirement is not exponential but proportional to the
>>  number of file saved. The CPU requirement is rawly proportional
>> to the volume of data to treat (CRC computation, compression, 
>> encryption, ...). This is true for both 64 bits and infinint 
>> flavors, though the infinint flavor does not rely on CPU integer 
>> operation where from its slowness.
> 
> 
> 
> Thank you for the clarification.  I didn't do the math over
> archives of different sizes, and went by my initial impression.

what I mentioned is theory. If system starts swapping the performance
degrades even faster while archive memory requirement increases, of
course...

> 
>>> The dar website seems to suggest that the cost of infinint is 
>>> modest, but my testing indicates that for what would be a 
>>> regular backup scenario, the cost is high.
> 
>> Where have you read that? This should be an error to be fixed.
> 
> 
> In the FAQ about a 'slight' penalty.
> 
> http://dar.linux.free.fr/doc/FAQ.html
> 
> Under "What slice size can I use with dar?" it says "thanks to its 
> internal own integer type named "infinint" dar is able to handle 
> arbitrarily large integers. This has a slightly memory and CPU
> penalty in regard to using native computer 32 or 64 bits integers,
> but has the advantage to provide a long term implementation in
> dar."

I will fix that, this is not correct, thanks for feedback.

[...]
> 
>>> It add in some cases unacceptable costs for no practical gain.
>>>  While some distributors compile with 64 bit integers (MacOSX 
>>> brew), other use the default (Fedora) which leads to a dar
>>> binary which people may consider broken or buggy.
> 
>> Well, that's correct...
> 
> 
> My first impression was a bug, and I was looking to migrate away
> from using dar until I did some web searching and thought that
> maybe the integer type was more significant than it might appear.
> I was a little concerned it might turn people off using dar if they
> find the program seems to hang for 10 minutes or so while restoring
> a single file .

I will probably follow your suggestion for release 2.6.0...

> 
> I'm not sure under what circumstances anyone would reach the
> stated limits though, unless I'm reading the website wrong and the
> file size limits are not 18EB but smaller.

you are reading correctly, limit can will be reached on any system if
the archive size (even split in many smaller slices) reaches 18EB.

I got positive feedback of several hundred TB archives, I think some
other guys reached the petabyte for their need some years ago, so we
are still far from the 18 exabyte... :-)

[...]

Cheers,
Denis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIVAwUBW22AZQgxsL0D2LGCAQip8A/+MaaD8ROdfv+jeCXlXisu+9cZws2v8IAz
F5bwPejEx1KMAXhIskeVnN0VOkiTj68pENeO9i34NHKCS3gCnUi+FJMnY48D5c7i
zReoRnIjFDIdZjNpresQyEFwEm91/+vucEA18WA38ZjEjHAi3kXUr2jpj42wGVby
YOX88lFXw0/5kWpvcMUf9h4H0y5HhKnfB8BOoxB4cbKO2xI5BhOD6+Df6Cso1iY2
pMXdxz9guYkjVoVhVe60+O9sKu5I0F8+iWuWOTUWh8s4kvJITxkCzZxeuHxYmXF5
XxFYR7KgZ8rwOxabQi0cvA62J+Mvi/GHPwhwGTLuo+qRI8HUz2y+5XOIBl3cfrnx
n6o5lFxl6X02k4AQoUOQ/isFUpNb7HEwm8ef2igLSLOiT9IKzJUALHe2IN1oj5G/
Ra7S0ixUViWCeHK1uuK2kk28AnJQY9Wn6eh1562j+ZZNmX3yVrDcQ44llQlHy+rr
ysmKNKwvtVP9V62HpAFUie+XtFiu9ZVK0jqo9vLB+huWsoQc0KnQi6mz3G9rMaV4
g+cnfg+STAY74pQvhalcrdtusGH//c/9BJfjYEN9LG/VFSonIsWyqIrLRIkCOo3e
3+iBeyisDrp9R8C3Bya1m1TtCcB7fh0EMxdPVg6iWuLH/RwjiyRX9gP53LLwMouN
7jB+WVoFJCg=
=H5f3
-----END PGP SIGNATURE-----