implement actualtext, artifact from recent graphicx

[xworld21]

Fix #2679 more or less as we discussed, but with more care about how the keys interact with each other:

• if artifact is true, @description is set to empty unless actualtext, alt are present
• if actualtext and alt are both present, save both as respectively description and aria-description

The latter might be slightly controversial. I played quickly with VoiceOver and it reads both alt and aria-description in sequence, with a proper pause (feels like "actualtext: alt"), which seems reasonable.

We could add a further heuristic that if actualtext and alt are equal, then just @description is enough.

I have added a test with all 8 combinations to clarify the intended semantics.

[u-fischer]

ngpdf (which implements the derivation from pdf to html) uses a <span> and drops the graphic if an actualtext is present, and <img alt=...> if there is an alt (but no actualtext) and ignores graphics with artifact. You can test that at https://texlive.net/ngpdf

\DocumentMetadata{tagging=on}
\documentclass{article}
\usepackage{graphicx}
\begin{document}
\includegraphics[actualtext=A,width=10pt]{example-image-a}t the begin there was

\includegraphics[alt=A duck]{example-image-duck}

\includegraphics[alt=A duck,actualtext=D]{example-image-duck}uck

\includegraphics[artifact,width=1cm]{example-image}
\end{document}

[brucemiller]

Thanks @u-fischer for the clarifications; that really helps! It's perhaps quite fortuitous that we're also dealing with a related issue (ACM's \Description) in #2666. It would be good to have a consistent, flexible approach to these data that we can build upon (and in particular, noting that HTML is not the only target).

Both these systems potentially supply material for two distinct use cases: (1) replacement text to be spoken in place of a given image or construct or (2) descriptive text used when a reader asks for more information. Of course, that distinction will easily be confused, and the way that the material is best delivered to the reader will depend on the eventual medium or delivery device. The artifact flag presumably just signals that the replacement text used should be empty (and not otherwise inferred from any other data).

And, although some of the documentation for these encourage using concise language free of markup, authors will do what they want, so I'm more inclined to store this data in XML elements, rather than attributes; and in particular, I'd want to avoid diving directly into aria and defer that until postprocessing (or XSLT) when the target format is dealt with.
Normally, I'd use the ltx:note element, with appropriate @role attribute, to distinguish the use cases, but am open to introducing new element(s) if that's more appropriate. (As @dginev will know, the ltx:note element is already quite heavily overloaded!)

[dginev]

As much as I appreciate the broader discussion on related work (thank you for raising awareness!), I am a little worried we'll hurt the primary use case without good cause.

Citing the graphicx documentation:

actualtext ActualText text for accessibility uses. By default this key is not
used but users are encouraged to add text here that may be used in tagged PDF or as the text to use in formats that do not use the image.

artifact Boolean to mark the graphic as an artifact for accessibility uses. By default this key is not used but users are encouraged to flag decorative or other non structural images as an artifact for tagged PDF.

The ngpdf implementation seems to be in conflict with the actualtext description (to me), as HTML as a format uses images, very similarly to PDF. I think keeping the image and adding an accessible label fits more naturally. As to handling artifact, mapping that to aria-hidden="true" seems to be a best practice as well from an ARIA perspective.

I am quite happy with Vincenzo's approach - there is a lot of work already done by the ARIA group which is clearly documented and standard. It will be great if we can use that without reinventing the full breadth of their spec with alternatives in LaTeXML's schema.

We can also separately create XSLT rules mapping any attributes in the aria: namespace to reasonable JATS <annotation>, <alt-text> etc elements, where appropriate. A good question to ask is "which spec is leading the adoption momentum" and whether we want to be in synergy with them. Adding a new custom schema piece here doesn't seem warranted to me, as @brucemiller has heard me opine before.

[u-fischer]

@dginev

The ngpdf implementation seems to be in conflict with the actualtext description (to me),

No, the graphicx documentation doesn't try to explain when to use the keys, it only says that the keys exist and that their value will be passed to PDF keys (alt=> /Alt, actualtext=>/ActualText) if a PDF is tagged. More documentation about how and when to use them is in the documentation of the tagging project.

ngpdf then handles these keys and it does it as specified in the derivation algorithm.

[dginev]

No, the graphicx documentation doesn't try to explain when to use the keys,

Well, the text I read exactly explains the intent of when to use the keys. What am I missing? I understand there are direct mappings for accessible PDF, but that isn't a supported target of LaTeXML.

[u-fischer]

The description only says "ActualText text for accessibility uses", which doesn't say much unless you know what the PDF term ActualText means.

The actual documentation about the intent of the keys is in latex-lab-graphic.pdf

[dginev]

The key snippet from the description is "or as the text to use in formats that do not use the image."

That is also extremely standard - text alternatives are needed when the reader doesn't consume the image for some reason or another. But most of latexml's target formats can use the underlying image asset. So they should, in my opinion.

[u-fischer]

The key snippet from the description is "or as the text to use in formats that do not use the image."

Yes, but that too is not very specific as it doesn't say which format are used.
The key has by purpose not a definitive description in this documentation as different formats can use it in different ways.

How you handle the keys is really up to you, but do not take alternative text and actualtext as two words for the same thing. In PDF there is a clear distinction and PDF is here different to html. Users do not expect to hear "graphic" if actualtext is used, see e.g. nvaccess/nvda#18843.

[brucemiller]

Addressing several points....
I find it rather unfortunate for a project like LaTeXML, that attempts to target different formats, when enhancements to LaTeX with broad utility are defined in terms of a single target. That said, given the general intention of these enhancements, I see no need for LaTeXML to actually discard the image when ActualText is given, so long as it can arrange that the actualtext is what's spoken when the document is being spoken accessibly, irrespective of what any other implementation may do.

Next, aria is indeed quite complex, and I wouldn't want to reinvent a whole alternative representation for it within LaTeXML. On the other hand, exactly because of that, if we've already cast the data into aria I have zero expectation that we'll be able to deconstruct it afterwards to reassemble into some other representation such as JATS. Thus I'd prefer a more abstract representation at the earlier, XML, stage. Experience with too early insertion of too much CSS would seem to have been a lesson.
Finally, that both \includegraphics alt keyword and \Description should provide a "description", but are here getting two distinct representations in our XML bothers me.

[u-fischer]

when enhancements to LaTeX with broad utility are defined in terms of a single target.

PDF is a major output format of LaTeX, and so it is unavoidable that there are keys and commands that target mostly PDF specific needs, e.g. pdftitle or pdflang from hyperref, or pdfversion and pdfstandard in \DocumentMetadata, or as here the artifact and actualtext key.

For such keys other output formats will have to decide if they ignore them or map to something in their format.

Finally, that both \includegraphics alt keyword and \Description should provide a "description", but are here getting two distinct representations in our XML bothers me

well \Description is a acmart specific command. Currently it doesn't seem to be used anywhere. You could open an issue and discuss with the maintainer how he plans to use it.

[brucemiller]

when enhancements to LaTeX with broad utility are defined in terms of a single target.

PDF is a major output format of LaTeX, and so it is unavoidable

I completely understand & agree; it's unavoidable there's going to be pdf specific data in LaTeX. My point was that some things (like accessibility), the data are not pdf-specific (though the implementation will be); in those cases avoiding too much pdf specificity in the documentation would be good (for users, implementers, converters & latex longevity in general).

well \Description is a acmart specific command. Currently it doesn't seem to be used anywhere. You could open an issue and discuss with the maintainer how he plans to use it.

Apparently authors are using it on arXiv; I'd be surprised if the maintainer hasn't got some plan for tying it into your pdf-accessibility machinery if not already. But my reason for mentioning it here was purely from a LaTeXML POV. Its intention as an object "description" (rather than speech replacement) seems clear, but I'm trying to assure that "descriptions", whatever package/binding they come from, are represented & treated consistently within LaTeXML.

[xworld21]

This generated more discussion than I expected!

I see two sticking points:

• Unlike with ngpdf.com, the HTML output of LaTeXML is supposed to be a complete PDF replacement, so it needs to include the picture. However, HTML does not have a mechanism comparable /ActualText (whose behaviour I read from (summaries of) the PDF spec itself, not from the graphicx documentation). There are some crude workarounds with background-image, none of them particularly good.
• This PR follows the naive HTML model and merges the actualtext=, alt= options. This is a lossy choise and prevents other outputs and downstream usesrs from making better choices.

My intention was to introduce initial support that can be merged in 0.9 so I will still campaign for merging as is, but I agree that the XML should preserve the info. I'd rather use additional attributes and/or children of the <ltx:graphics> element though. /ActualText and /Alt are pure text in PDF, so they don't need to be elements; and even if they were elements like <ltx:note>, they should be descendants of <ltx:graphics> both for logical and practical XSLT reasons.

[brucemiller]

I'll take the view that the discussion only sounds more contentious than it really is.

I'm not sure I understand @xworld21 first sticking point; Don't the two pieces of information we're discussing just map to aria's description (for alt and \Description) or label (for actualtext)? [when aria is applicable].
Of course, historically html img's @alt represents both those cases and \includegraphics[alt] as well as LaTeXML's ltx:graphics@description have inherited that blur. It's kinda amusing that your code is forced to use two attributes, @description and @aria:description, for the 2 distinct quantities! So, it would be good (if attributes are the right choice) to have an attribute to use for the replacement text (but label is not a very good name in a LaTeX context).

So for me, the sticking point is whether I can convince yall that using elements for these two (which would need to be within the meta-class and be allowed content within an ltx:graphics), and the collect the conversion of these into a single place in postprocessing that would convert them to appropriate aria, whenever aria is the right choice.

[dginev]

Changing to using an element to hold the accessible text, giving it an xml:id, then pointing to it via an aria:labelledby or aria:describedby is fine by me. If the #2666 PR is merged we will have a simple ltx:note formulation that can do that.

Reinventing a custom LaTeXML technique to do the pointing just doesn't strike me as helpful. We'll increase the learning curve and confuse anyone external about what LaTeXML is trying to do differently. I am quite happy we are using the xml:id attribute and not a ltx:xmlid mirror, for example.

[brucemiller]

Shouldn't this be aria:label?
Perhaps you were thrown by the awkward naming of ltx:graphics@description (as I was), which is a stand-in for html's img@alt, whose specification seems to shift over time.
The graphics keywords alt and actualtext are apparently only very subtly different; there's not (yet) a key for a detailed description, which is what aria:description should be (as I understand it).

[brucemiller]

Hopefully we've all had a chance to recover from that exhausting debate. And if we focus on actualtext,artifact, as @xworld21 had intended (leaving the confusions of long, short and long-short descriptions to a separate debate), the PR here is actually quite reasonable and reasonably scoped.

That said, I think the current code goes a bit too far to preserve everything, whereas I'm inclined to think that the intentions of actualtext, artifact, and alt are largely exclusive (ie. you really should only use one of them):

• artifact signals non-textual, decorative elements that should be ignored by screen readers and removed from the "tag tree"; thus, any description or actualtext should be irrelevant. So, I think that artifact should generate aria:hidden='true', and any alt or actualtext keys given should be ignored (possibly with an Info-level warning). We probably should set the description attribute to empty string for validation purposes, though.
• actualtext provides replacement text for text rendered in a way that screen readers cannot interpret, such as illuminated initials, dropcaps and such; the actualtext would normally be very short, maybe a single letter. The goal with alttext="foo", I think, is that the reader would say "foo" (or even ideally blend it into the adjacent text?), not "graphic foo" or similar. I'm no aria expert, but I think this means we should set aria:label, rather than aria:description. Again, any alt key should be ignored (or warned). And again, it is probably best for validation to also set the description attribute to the actualtext, for validation.
• alt is otherwise fine, for the moment: As I said, we can leave alt, and questions of attributes vs elements to another day, another debate.

Does this sound like a reasonable interpretation?

[xworld21]

That said, I think the current code goes a bit too far to preserve everything, whereas I'm inclined to think that the intentions of actualtext, artifact, and alt are largely exclusive (ie. you really should only use one of them):

I agree with the outcome

• artifact => aria-hidden
• actualtext => @aria-label
• alt => @description

It's not a perfect translation, but it's lossless and less convoluted than my code.

Now I have just read bits of the PDF specs, and the whole thread pdf-association/pdf-issues#60, and... I still don't understand what we should do here. At any rate, they don't seem mutually exclusive except maybe for artifact.

My final takeaway, though, is that /ActualText has no equivalent in HTML, at all (it is a 'character replacement', which is just not something you can do). Moreover, to my memory @aria-label doesn't work on <img>, plus we don't have @alt on <object>. I'll have to test a few screen readers again and reread the relevant specs, but most likely I'll attach an XSLT change to use @alt no matter which attribute was passed (or we reproduce my trick of shifting @description to aria-description so it's not lost).

[brucemiller]

Thanks for the link to pdf discussion, although I also find it somewhat perplexing. They never say that these 3 are exclusive, but it's hard for me to imagine what would be expected when there are more than 1.

But for me the takeaway is that actualtext is a replacement, not a description. I wasn't aware that aria-label doesn't work on img (and am not sure I even care anymore? it seems as close as we can reasonably get).
Well, actually I can concoct a "solution":

<span aria-label="Hello"><img src="https://github.com/fancyh.png"/>ello</span>

but it's too brittle to try to implement in general; Would be easy for something like the lettrine package, though.

I'm not sure I understood your last point about @alt. My thinking was that for the artifact and actualtext cases, we'd set @description to either empty or the actualtext only in order to set the img@alt attribute to keep validators happy; it is not really a description. (perhaps more strict: we could omit @description and have the XSLT add @alt if it recognizes the aria hidden or label?). Either way, we should not add aria-description, since there is no description in these cases.

In the more common cases of @alt (or acm's \Description), I'd prefer to defer how we'll deal with longer descriptions, possibly with markup, new element, etc to a discussion about (perhaps) aria-describedby.

[dginev]

I will just pass by to remark that in this rare moment when we have consensus, it is a great time to act before the winds blow again and we find out there is more to discuss... 🍃

Stretch goals can always come after...

[xworld21]

I will just pass by to remark that in this rare moment when we have consensus, it is a great time to act before the winds blow again and we find out there is more to discuss... 🍃

Right! Let's get the XML first, we can discuss HTML and ARIA in a separate, more specific PR. Expect an update this weekend.

[xworld21]

ARIA reference https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Reference/Roles/img_role#associated_wai-aria_roles_states_and_properties

So I have mapped the attributes to @aria:label and @description, like we said. However, given that <img> is special (see link), I have modified the stylesheet to use the @alt tag instead of @aria-label, and shift @description to @aria-description when @alt is already being used. I have also removed the caption references when the image is an artifact or has an actual text, as that was always a fallback, and it's clearly not necessary in that case.

By the way, I added role="img" for SVG. Apparently <object> has no implicit role so it's best to specify one.

I still think that the better solution is to preserve the LaTeX intent in XML, so use attributes @artifact, @actualtext, and let postprocessing deal with it according to the output (e.g. JATS and TEI don't understand ARIA, do they?).