fix_1402_rafimage_0.27 #1409

clanmills · 2020-11-24T20:53:37Z

WIP #1402 rafimage::printStructure() improved formatting.

lgtm-com · 2020-11-24T21:06:45Z

This pull request introduces 12 alerts when merging 76514a1 into 4af8b9b - view on LGTM.com

new alerts:

12 for Wrong type of arguments to formatting function

lgtm-com · 2020-11-25T16:53:46Z

This pull request fixes 6 alerts when merging 45119e3 into 52fd70a - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

lgtm-com · 2020-11-26T13:47:01Z

This pull request fixes 6 alerts when merging c48189a into 12e6307 - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

clanmills · 2020-11-26T20:40:23Z

Old behaviour:

1018 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $ exiv2 -g Fujifilm ~/Downloads/sample1.raf  
Exif.Fujifilm.Version                        Undefined   4  48 49 51 48
Exif.Fujifilm.SerialNumber                   Ascii      48  FF02B4550110     593534373734180817925330110168
Exif.Fujifilm.Quality                        Ascii       8  NORMAL 
Exif.Fujifilm.Sharpness                      Short       1  Normal
Exif.Fujifilm.WhiteBalance                   Short       1  Auto
Exif.Fujifilm.Color                          Short       1  Normal
Exif.Fujifilm.FlashMode                      Short       1  (38976)
Exif.Fujifilm.FlashStrength                  SRational   1  0/100
Exif.Fujifilm.FocusMode                      Short       1  Auto
Exif.Fujifilm.SlowSync                       Short       1  Off
Exif.Fujifilm.PictureMode                    Short       1  Aperture-priority AE
Exif.Fujifilm.ShadowTone                     SLong       1  0
Exif.Fujifilm.HighlightTone                  SLong       1  0
Exif.Fujifilm.Continuous                     Short       1  Off
Exif.Fujifilm.SequenceNumber                 Short       1  0
Exif.Fujifilm.BlurWarning                    Short       1  Off
Exif.Fujifilm.FocusWarning                   Short       1  Off
Exif.Fujifilm.ExposureWarning                Short       1  Off
Exif.Fujifilm.DynamicRange                   Short       1  Standard
Exif.Fujifilm.FilmMode                       Short       1  F0/Standard (Provia)
Exif.Fujifilm.DynamicRangeSetting            Short       1  Raw
Exif.Fujifilm.DevelopmentDynamicRange        Short       1  100
Exif.Fujifilm.MinFocalLength                 Rational    1  1800/100
Exif.Fujifilm.MaxFocalLength                 Rational    1  5500/100
Exif.Fujifilm.MaxApertureAtMinFocal          Rational    1  280/100
Exif.Fujifilm.MaxApertureAtMaxFocal          Rational    1  400/100
Exif.Fujifilm.Rating                         Long        1  0
1019 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $

New behaviour:

1016 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exiv2 -g Fujifilm ~/Downloads/sample1.raf 
Exif.Fujifilm.Version                        Undefined   4  48 49 51 48
Exif.Fujifilm.SerialNumber                   Ascii      48  FF02B4550110     593534373734180817925330110168
... as above 
Exif.Fujifilm.Rating                         Long        1  0
... and now the beautiful tags from the embedded tiff ...
Exif.Fujifilm.FujiIFD                        Ifd         1  26
Exif.Fujifilm.RawImageFullWidth              Long        1  6384
Exif.Fujifilm.RawImageFullHeight             Long        1  4182
Exif.Fujifilm.BitsPerSample                  Long        1  14
Exif.Fujifilm.StripOffsets                   Long        1  2048
Exif.Fujifilm.StripByteCounts                Long        1  53395776
Exif.Fujifilm.BlackLevel                     Long       36  1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022
Exif.Fujifilm.GeometricDistortionParams      SRational  19  3749/9 502/1420 710/1420 870/1420 1004/1420 1123/1420 1230/1420 1328/1420 1420/1420 1506/1420 35789/65536 64000/65536 92105/65536 121578/65536 152315/65536 183578/65536 216947/65536 251473/65536 286000/65536
Exif.Fujifilm.WB_GRBLevelsStandard           Long        8  302 371 826 17 302 626 485 21
Exif.Fujifilm.WB_GRBLevelsAuto               Long        3  302 561 552
Exif.Fujifilm.WB_GRBLevels                   Long        3  302 561 552
Exif.Fujifilm.ChromaticAberrationParams      SRational  29  3749/9 502/1420 710/1420 870/1420 1004/1420 1123/1420 1230/1420 1328/1420 1420/1420 1506/1420 4/65536 2/65536 -2/65536 -6/65536 -8/65536 -12/65536 -14/65536 -18/65536 -18/65536 -58/65536 -54/65536 -50/65536 -46/65536 -40/65536 -32/65536 -24/65536 -16/65536 -16/65536 3749/9
Exif.Fujifilm.VignettingParams               SRational  19  3749/9 502/1420 710/1420 870/1420 1004/1420 1123/1420 1230/1420 1328/1420 1420/1420 1506/1420 192752/2048 183960/2048 176751/2048 171126/2048 158275/2048 138976/2048 119700/2048 101213/2048 87673/2048
1017 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exiv2 -g Fujifilm ~/Downloads/sample1.raf  | grep Bits
Exif.Fujifilm.BitsPerSample                  Long        1  14
1018 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $

Delighted to say that the embedded tags are preserved in the EXV:

1020 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exiv2 -ea --verbose --force ~/Downloads/sample1.raf  
File 1/1: /Users/rmills/Downloads/sample1.raf
Writing Exif data from /Users/rmills/Downloads/sample1.raf to /Users/rmills/Downloads/sample1.exv
1021 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $ bin/exiv2 --grep Fujifilm ~/Downloads/sample1.exv
Exif.Fujifilm.Version                        Undefined   4  48 49 51 48
Exif.Fujifilm.SerialNumber                   Ascii      48  FF02B4550110     593534373734180817925330110168
Exif.Fujifilm.Quality                        Ascii       8  NORMAL 
Exif.Fujifilm.Sharpness                      Short       1  Normal
Exif.Fujifilm.WhiteBalance                   Short       1  Auto
Exif.Fujifilm.Color                          Short       1  Normal
Exif.Fujifilm.FlashMode                      Short       1  (38976)
Exif.Fujifilm.FlashStrength                  SRational   1  0/100
Exif.Fujifilm.FocusMode                      Short       1  Auto
Exif.Fujifilm.SlowSync                       Short       1  Off
Exif.Fujifilm.PictureMode                    Short       1  Aperture-priority AE
Exif.Fujifilm.ShadowTone                     SLong       1  0
Exif.Fujifilm.HighlightTone                  SLong       1  0
Exif.Fujifilm.Continuous                     Short       1  Off
Exif.Fujifilm.SequenceNumber                 Short       1  0
Exif.Fujifilm.BlurWarning                    Short       1  Off
Exif.Fujifilm.FocusWarning                   Short       1  Off
Exif.Fujifilm.ExposureWarning                Short       1  Off
Exif.Fujifilm.DynamicRange                   Short       1  Standard
Exif.Fujifilm.FilmMode                       Short       1  F0/Standard (Provia)
Exif.Fujifilm.DynamicRangeSetting            Short       1  Raw
Exif.Fujifilm.DevelopmentDynamicRange        Short       1  100
Exif.Fujifilm.MinFocalLength                 Rational    1  1800/100
Exif.Fujifilm.MaxFocalLength                 Rational    1  5500/100
Exif.Fujifilm.MaxApertureAtMinFocal          Rational    1  280/100
Exif.Fujifilm.MaxApertureAtMaxFocal          Rational    1  400/100
Exif.Fujifilm.Rating                         Long        1  0
Exif.Fujifilm.RawImageFullWidth              Long        1  6384
Exif.Fujifilm.RawImageFullHeight             Long        1  4182
Exif.Fujifilm.BitsPerSample                  Long        1  14
Exif.Fujifilm.StripOffsets                   Long        1  2048
Exif.Fujifilm.StripByteCounts                Long        1  53395776
Exif.Fujifilm.BlackLevel                     Long       36  1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022 1022
Exif.Fujifilm.GeometricDistortionParams      SRational  19  3749/9 502/1420 710/1420 870/1420 1004/1420 1123/1420 1230/1420 1328/1420 1420/1420 1506/1420 35789/65536 64000/65536 92105/65536 121578/65536 152315/65536 183578/65536 216947/65536 251473/65536 286000/65536
Exif.Fujifilm.WB_GRBLevelsStandard           Long        8  302 371 826 17 302 626 485 21
Exif.Fujifilm.WB_GRBLevelsAuto               Long        3  302 561 552
Exif.Fujifilm.WB_GRBLevels                   Long        3  302 561 552
Exif.Fujifilm.ChromaticAberrationParams      SRational  29  3749/9 502/1420 710/1420 870/1420 1004/1420 1123/1420 1230/1420 1328/1420 1420/1420 1506/1420 4/65536 2/65536 -2/65536 -6/65536 -8/65536 -12/65536 -14/65536 -18/65536 -18/65536 -58/65536 -54/65536 -50/65536 -46/65536 -40/65536 -32/65536 -24/65536 -16/65536 -16/65536 3749/9
Exif.Fujifilm.VignettingParams               SRational  19  3749/9 502/1420 710/1420 870/1420 1004/1420 1123/1420 1230/1420 1328/1420 1420/1420 1506/1420 192752/2048 183960/2048 176751/2048 171126/2048 158275/2048 138976/2048 119700/2048 101213/2048 87673/2048
1022 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance/build $

Tomorrow I will add a test in the test suite>

@FreddieWitherden please test with your files and give me feedback.

I'll ask for Team Review after I've heard back from Freddie.

lgtm-com · 2020-11-26T20:42:54Z

This pull request fixes 6 alerts when merging e032586 into 0b5ba68 - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

FreddieWitherden · 2020-11-27T01:53:04Z

This works an absolute treat for me on all of my files. I will post some examples of what this extra data allows me to do tomorrow morning.

A couple of queries. Firstly, do you think there is any safe way we could avoid reading the entire TIFF, which can be tens of megabytes? For example, perhaps reading 64k, and if that returns an exception, retry by reading the entire TIFF? This may help in the common case where the metadata is near the start. Secondly, should we consider wrapping the new code in an try/catch or other error handling block to ensure that all old files will still work (even if the additional TIFF we are reading is broken, say).

clanmills · 2020-11-27T14:07:30Z

I've updated the test harness and requested @piponazo to review the code.

I was puzzled when the following metadatum was not in the .EXV file. That's correct. The Exif serialiser doesn't know how to create sub-ifds. That would be a rather complex change in class TiffWriter. I don't think that's worth the effort to fix.

Exif.Fujifilm.FujiIFD                        Ifd         1  26

If everything is working for @FreddieWitherden, it's time to move on. I'm working on my book and don't want to spend more time on this issue. If RAF files surface that require more cautious handling of the embedded TIFF, a new issue should be opened and investigated.

lgtm-com · 2020-11-27T14:08:23Z

This pull request fixes 6 alerts when merging 3ffd807 into 0b5ba68 - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

clanmills · 2020-11-27T16:24:52Z

Thanks for approving the change, @FreddieWitherden . When the CI goes green, I'll merge. It has been fun to work with you again.

FreddieWitherden · 2020-11-27T16:41:44Z

To showcase what these additional tags let enable below is a photo I took last night of a whiteboard in my office. Observe how it appears to bulge out and is badly warped. Also note how dark the corners are.

Thankfully these tags contain correction coefficients. Using them we obtain the image below.

Which is a dramatic improvement and much closer to the out of camera JPEG's.

clanmills · 2020-11-27T16:51:52Z

Wow. That's a huge improvement. Presumably you're corrected that using Exif.Fujifilm.GeometricDistortionParams.

I see some Visual Studio builds have failed. I'll investigate and fix that before merging.

FreddieWitherden · 2020-11-27T16:58:20Z

Yes, so pulling out those parameters for my photo we have:

416.5555556
0.3535211268 0.5 0.6126760563 0.7070422535 0.7908450704 0.8661971831 0.9352112676 1 1.06056338
-1.543533325 -3.429199219 -5.157470703 -6.749191284 -8.312011719 -9.669250488 -10.78233337 -11.68019104 -11.68019104

Now, I am unsure what the first parameter corresponds to, but the next 9 are locations (normalised such that 0 is the image centre and 1 is the corner), and the following 9 are the amount of distortion in % terms are these points. So at the far corner we have -11.7% (!) distortion. With this it information to hand it isn't too difficult to correct the image.

clanmills · 2020-11-27T17:03:22Z

And you fixed the image in what? darktable ?

The new test has failed on Visual Studio. How weird. I'll investigate and fix it.

Here's the bill for my help. Can you go to https://openhub.net (free to register). Give me kudos and say something nice about Exiv2.

clanmills · 2020-11-27T18:43:29Z

The CI builds are failing on the new test on both Visual Studio and Linux. I don't have a problem with it when I build on Ubuntu, Visual Studio 2019 and macOS on my machines. I'll make an insignificant change to trigger another build on the CI.

lgtm-com · 2020-11-27T19:02:32Z

This pull request fixes 6 alerts when merging a71eb24 into 54506bd - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

clanmills · 2020-11-27T19:58:00Z

This failure is being caused by the CI outputting:

Exif.Fujifilm.CropMode                       Short       1  None

And my machines are outputting:

Exif.Fujifilm.Continuous                     Short       1  Off

How can this be? There's a way to hide the issue by using -pv to avoid interpreting tag 0x1100. That's not a good way to go. I want to understand how/why the CI outputs "CropMode".

Tomorrow is another day!

FreddieWitherden · 2020-11-27T20:01:19Z

Aha!

I think this is because post-merge of this PR #1407 will be included, whereas at the moment this branch does not have #1407 backported, hence the difference in output.

clanmills · 2020-11-27T21:49:25Z

Two Things are Terrific Together - Testing and Team-work.

Somehow, your CropMode fix is in conflict. I'll modify my test to expect CropMode and all will be well (eventually). You'll visit https://openhub.net and thank me for a week's work on your behalf!

I would like to thank you for bringing this whole matter up. I've dealt with RAF in the book and discovered new/hidden secrets of the Exiv2 code base. The purpose of my book is to enable a future for Exiv2 when I retire. Working with you has strengthened the book. Thanks.

lgtm-com · 2020-11-27T22:20:52Z

This pull request fixes 6 alerts when merging a1dca06 into 54506bd - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

clanmills · 2020-11-28T09:12:37Z

Thanks for giving me Kudos on OpenHub.

Something horrible is happening. The CI is now reporting Continuous and test has CropMode. Here's what I see:

1116 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ git checkout 0.27-maintenance
Switched to branch '0.27-maintenance'
Your branch is up to date with 'origin/0.27-maintenance'.
1117 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ grep 1100 src/fuj*.cpp
    //! Continuous, tag 0x1100
        TagInfo(0x1100, "Continuous", N_("Continuous"),
1118 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ git checkout fix_1402_rafimage_0.27
Switched to branch 'fix_1402_rafimage_0.27'
Your branch is up to date with 'origin/fix_1402_rafimage_0.27'.
1119 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $ grep 1100 src/fuj*.cpp
    //! Continuous, tag 0x1100
        TagInfo(0x1100, "Continuous", N_("Continuous"),
1120 rmills@rmillsmm-local:~/gnu/github/exiv2/0.27-maintenance $

It's a total coincidence that CropMode and Continuous are neighbours in the code and in the metadata of the test images. I've applied #1407 into this PR and I think that will calm everybody down.

Tag 0x1100 is AutoBracketing which Exiv2 calls "Continuous"

Tag 0x104d is CropMode.

lgtm-com · 2020-11-28T09:27:07Z

This pull request fixes 6 alerts when merging 2466c78 into 54506bd - view on LGTM.com

fixed alerts:

6 for Wrong type of arguments to formatting function

WIP #1402 rafimage::printStructure() improved formatting.

76514a1

clanmills added this to the v0.27.4 milestone Nov 24, 2020

clanmills self-assigned this Nov 24, 2020

clanmills linked an issue Nov 24, 2020 that may be closed by this pull request

Support for FujiFilm IFD Tags #1402

Closed

clanmills marked this pull request as draft November 24, 2020 20:54

WIP: working to understand how to support tiffIfd in tiffvisitor_int.cpp

45119e3

clanmills added 2 commits November 26, 2020 13:28

Remove debugging code.

a8b549f

Add FujiIFD to TiffCreator::tiffGroupStruct_

c48189a

clanmills added 2 commits November 26, 2020 20:23

Success. It's working!

9b877f8

Code/comment tidy.

e032586

clanmills mentioned this pull request Nov 26, 2020

Support for FujiFilm IFD Tags #1402

Closed

Add to test harness.

3ffd807

clanmills marked this pull request as ready for review November 27, 2020 13:56

clanmills requested a review from piponazo November 27, 2020 13:56

FreddieWitherden approved these changes Nov 27, 2020

View reviewed changes

clanmills removed the request for review from piponazo November 27, 2020 16:23

Pointless change to trigger CI to build again.

a71eb24

Hoping for CI Contentment!

a1dca06

Calming Test Suite concerning Continuous and CropMode confusion.

2466c78

clanmills merged commit 866d213 into 0.27-maintenance Nov 28, 2020

clanmills deleted the fix_1402_rafimage_0.27 branch November 28, 2020 09:40

clanmills mentioned this pull request Nov 28, 2020

Current status of bash tests on master #1413

Closed

clanmills mentioned this pull request Jan 11, 2021

Exiv2 RoadMap #1018

Open

fix_1402_rafimage_0.27 #1409

fix_1402_rafimage_0.27 #1409

Uh oh!

Conversation

clanmills commented Nov 24, 2020

Uh oh!

lgtm-com bot commented Nov 24, 2020

Uh oh!

lgtm-com bot commented Nov 25, 2020

Uh oh!

lgtm-com bot commented Nov 26, 2020

Uh oh!

clanmills commented Nov 26, 2020

Uh oh!

lgtm-com bot commented Nov 26, 2020

Uh oh!

FreddieWitherden commented Nov 27, 2020

Uh oh!

clanmills commented Nov 27, 2020

Uh oh!

lgtm-com bot commented Nov 27, 2020

Uh oh!

clanmills commented Nov 27, 2020

Uh oh!

FreddieWitherden commented Nov 27, 2020

Uh oh!

clanmills commented Nov 27, 2020

Uh oh!

FreddieWitherden commented Nov 27, 2020

Uh oh!

clanmills commented Nov 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clanmills commented Nov 27, 2020

Uh oh!

lgtm-com bot commented Nov 27, 2020

Uh oh!

clanmills commented Nov 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FreddieWitherden commented Nov 27, 2020

Uh oh!

clanmills commented Nov 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lgtm-com bot commented Nov 27, 2020

Uh oh!

clanmills commented Nov 28, 2020

Uh oh!

lgtm-com bot commented Nov 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clanmills commented Nov 27, 2020 •

edited

Loading

clanmills commented Nov 27, 2020 •

edited

Loading

clanmills commented Nov 27, 2020 •

edited

Loading