Start a new topic

Wikitude and OpenCV in Xamarin

Hi Wikitude and Users,


we can't get Wikitude working in a Xamarin crossplatform project (Android and iOS), which also uses OpenCV.


### The Setup


Xamarin crossplatform project using MvvmCross with the following setup:


* Core-project, PCL providing view models, business logic and platform-agnostic logic in general.

* OpenCV.Droid, bindings project providing OpenCV to the Android part.

    * Android Library (AAR), created from the official OpenCV SDK for Android.

* OpenCV.iOS, bindings project providing OpenCV to the iOS part using P/Invoke.

    * Own static library project.

    * Includes the static library from the official OpenCV SDK for iOS, which is just an Apple Framework (combining the static library and headers).

    * Exposes OpenCV functionality via own API to minimize the amount of marshaling code.

* UI.Droid, Android platform project providing views and platform specific logic.

* UI.iOS, iOS platform project providing views and platform specific logic.


Our ARchitectWorld uses Object Tracking and could successfully be tested in the Wikitude examples contained in the Xamarin Component for both Android and iOS.


### The Problem


When starting the ARchitectWorld in our iOS project, the world is loaded and we receive the `onLoaded`-callback when creating the `TargetCollectionResource`, as well as the `onTargetsLoaded`-callback when creating the `ObjectTracker`. So far, the behavior is the same in Wikitude's sample project, as well as our project.


The problem is, that the same object using the same WTO-file is not being recognized in our project, while it's working very well in the sample project.


### The Investigation


When that problem occurred, we streamlined the logic in the ViewController which handles the ARchitectWorld, to rule out problems in our code there.


We then investigated our project by adding the ARchitectWorld to different commits in our repo's history, which we deemed critical changes and found commits, were object tracking was working.


To find the critical change, we used git-bisect and identified the commit which introduced the problem. With that commit the static library of our OpenCV bindings project was updated, because it was missing the device-architectures - it only had the simulator archs, basically x86_64.


After that, we assumed, Wikitude might use OpenCV internally and we could get symbol collisions, even though the linker didn't complain about duplicate symbols. Though, Wikitude does not use OpenCV but a search for the combined use of Wikitude and OpenCV brought up some results without solutions:


* Crash when work with OpenCV

* Wikitude iOS JavaScript Crash on detection


So then we found out, Wikitude and OpenCV both include partly the same third party code.


### The questions


In the linked posts, the Wikitude support answered quite openly, suggesting an internal discussion and coming back to the OPs. Though no further answers are in the thread, suggesting either there is none or the information has been exchanged through another channel.


* Is there any information you, Wiktude support, can give us about this topic?

* Can try to help otherwise or do you have suggestions?


In other posts, we saw the suggestion of using OpenCV with Wikitude, by preparing visual data and feed it to an input plugin. This suggests, that OpenCV and Wikitude can be used together.


* Can, and if, how can this be achieved in a Xamarin project like the one I described?

* Does it require a certain version of OpenCV? (We are currently using v3.3.1.)


Very looking forward to a response ;-)


Regards,

Felix


We finally have a working application using Wikitude and OpenCV. Though we were not able to clearly identify the root cause. The steps that led to a working solution are as follows:


1. Build OpenCV from the source with only the modules we need using the `--without` parameter for the build script `build_framework.py` to reduce the amount of possible conflicts. This was wrong anyways, as the binaries get stripped during native linking and no errors appeared.

    * Doing this with the downloadable source of v3.3.1, the version of OpenCV we have been using, we ran into a symbol problem during native linking, when compiling the Xamarin project.

    * This led to an issue in OpenCV, that has just recently been fixed (2018-01-10): https://github.com/opencv/opencv/issues/10565  

        (We were not building with `--dyanmic`, still faced the error during native linking when compiling the Xamarin project.)

2. Building OpenCV from current master on GitHub (commit `17233c687e2d51010dab2587f2532a460ccdb4b6`, something beyond v3.4.0) turns out to work and not break Wikitude: https://github.com/opencv/opencv

3. Using the downloadable OpenCV Framework for iOS (v3.4.0) led to the same issue during native linking.


TL;DR


Building OpenCV from its source on GitHub, at least commit `17233c687e2d51010dab2587f2532a460ccdb4b6` turned out to not break Wikitude.


We will have a look at the downloadable iOS framework of the next release of OpenCV, to see, whether that works and return to using that, instead of building OpenCV ourselves.


Regards

Hi Felix,

From similar support requests we know that a potential root cause can be duplicate symbols of libraries such as jpeg. They might not fail the build but for one of the libraries a different symbols then expected is loaded/used at runtime, which then leads to unexpected behavior.

So maybe through the usage of the `--without`, you removed the OpenCV modules that rely on symbols that are used/provided by both libraries which then leads to a application that is ok.


Anyhow, I'm glad you figured out a way of using both libraries at the same time.


Best regards,

Andreas

Login or Signup to post a comment