Wikitude

If I understand the documentation here (https://www.wikitude.com/external/doc/documentation/6.0/unity/pluginsapiunity.html) the new Camera2 interface the Frame returned in Unity with the YUV_420_NV21 colorspace is ImageFormat.YUV_420_888. The actual organization of this format is unknown across devices (the UV plane may or may not be interleaved). In order to reliably convert this format to NV21 I need to know the pixel stride which is available in Android when receiving Images through the camera preview. However, the Wikitude Frame object provides no such information. As a result, I have no reliable way of converting the Frame data into NV21. For reference, see https://bugs.chromium.org/p/libyuv/issues/detail?id=604. Do I fail to understand the problem or am I misunderstanding the Wikitude documentation?

I would recommend that you convert the image to NV21 in native code before passing the bytes to Unity.

The Android specification for this format does not specify what order the U and V planes are ordered in. The documentation says it is "capable of describing any 4:2:0 chroma-subsampled planar or semiplanar buffer (but not fully interleaved)". This is not a significant issue when using the Java Image class because the U and V planes can be accessed directly (i.e. the 1 and 2 index of the planes is guaranteed to correspond to the U and V planes respectively). However, when the data has already been put into a single buffer if becomes impossible to know whether a UV interleaved plane consists of NV21 or NV12 ordering. We can live with having the stride information, we will cross our fingers that it will cover most situations, but the only practical way to do the conversion may be in native code utilizing the NEON chipset (as found in libyuv).  I realize that the Camera2 interface has created this issue but it remains a performance, efficiency and reliability concern that ideally should not be imposed upon consumers of the data (especially in a non-native environment like C#).

Can you clarify what you mean by "The format will always be Yv12."?

Are you saying that the format will always be NV12?

If so then we are reporting that we are not seeing interleaved UV values.

If you are saying that the format will always be a 12 format (i.e. U followed by V) then this resolves our concern about the ordering.

And, you might consider changing the "colorspace" enumerator to something other than YUV_420_NV21.

Sorry, but one final point of clarification.

The purpose of the original posting was to point out that developers have suggested that different devices may produce different results from the Camera2 interface (i.e. interleaved or non-interleaved UV).

Is it still the case (as your documentation states) that the result is the ImageFormat.YUV_420_888 format and therefore we will need the pixel stride information to determine whether it is interleaved?

Or is it "expected that the values are not interleaved" in which case we do not need the stride information?

I should clarify something from the comments above.

The format returned by Wikitude appears to be YV12 format which is V followed by U ( I stated incorrectly before).

This should not be confused with NV12 which is actually U followed by V or NV21 which is V followed by U (which is confusing indeed).