Xamarin.Android - Using Computer Vision API To Analyze Images Taken By Camera

 

Introduction

In this article, I will show you how to consume Computer Vision API in Xamarin.Android for analyzing the images captured by a mobile camera. I hope you will learn the latest concepts in Xamarin using Cognitive Services.

 

Prerequisites

• Computer Vision API Key
• Microsoft.Net.Http
• Newtonsoft.Json

Computer Vision API keys

 

 

Computer Vision services require special subscription keys. Every call to the Computer Vision API requires a subscription key. This key needs to be either passed through a query string parameter or specified in the request header.

 

To sign up for subscription keys, see Subscriptions. It's free to sign up. Pricing for these services is subject to change.

 

If you sign up using the Computer Vision free trial, your subscription keys are valid for the West-Central region (https://westcentralus.api.cognitive.microsoft.com).

 

The steps given below are required to be followed in order to create an Image analysis app in Xamarin.Android using Visual Studio.

 

Step 1 - Create an Android Project

 

Create your Android solution in Visual Studio or Xamarin Studio. Select Android and from the list, choose Android Blank App. Give it a name, like ImageAnalyzebyCamera.

 

(ProjectName: ImageAnalyzebyCamera)

 

Step 2 - Add References of NuGet Packages

 

First of all, in References, add the references to Microsoft.Net.Http and Newtonsoft.Json using NuGet Package Manager, as shown below.

 

 

Step 3 - User Interface

 

Open Solution Explorer-> Project Name-> Resources-> Layout-> Main.axml and add the following code. The layout will have an ImageView in order to display the preview of the sample image. I also added a TextView to display the contents of the Image.

 

(FileName: Main.axml)

1. <?xml version="1.0" encoding="utf-8"?>  
2. <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"  
3.     xmlns:app="http://schemas.android.com/apk/res-auto"  
4.     xmlns:tools="http://schemas.android.com/tools"  
5.     android:layout_width="match_parent"  
6.     android:layout_height="match_parent">  
7.     <ImageView  
8.         android:id="@+id/image"  
9.         android:layout_width="match_parent"  
10.         android:layout_height="match_parent"  
11.         android:layout_above="@+id/group_button" />  
12.     <LinearLayout  
13.         android:layout_above="@+id/txtDescription"  
14.         android:id="@+id/group_button"  
15.         android:layout_width="match_parent"  
16.         android:layout_height="wrap_content"  
17.         android:orientation="horizontal"  
18.         android:weightSum="2">  
19.         <Button  
20.             android:id="@+id/btnCapture"  
21.             android:layout_width="0dp"  
22.             android:layout_height="wrap_content"  
23.             android:layout_weight="1"  
24.             android:text="Capture" />  
25.         <Button  
26.             android:id="@+id/btnProcess"  
27.             android:layout_width="0dp"  
28.             android:layout_height="wrap_content"  
29.             android:layout_weight="1"  
30.             android:text="Analyze" />  
31.     </LinearLayout>  
32.     <TextView  
33.         android:id="@+id/txtDescription"  
34.         android:layout_width="match_parent"  
35.         android:layout_height="50dp"  
36.         android:text="Description: "  
37.         android:textSize="20sp"  
38.         android:layout_alignParentBottom="true" />  
39. </RelativeLayout>   

Step 4 - Analysis Model Class

 

Add a new class to your project with the name AnalysisModel.cs. Add the following properties to get the result set from JSON response with an appropriate namespace.

 

(FileName: AnalysisModel.cs) 

1. using System.Collections.Generic;  
2.   
3. namespace ImageAnalyze  
4. {  
5.     public class AnalysisModel  
6.     {  
7.         public IList<Category> categories { get; set; }  
8.         public object adult { get; set; }  
9.         public IList<Tag> tags { get; set; }  
10.         public Description description { get; set; }  
11.         public string requestId { get; set; }  
12.         public Metadata metadata { get; set; }  
13.         public IList<Face> faces { get; set; }  
14.         public Color color { get; set; }  
15.         public ImageType imageType { get; set; }  
16.     }  
17.     public class FaceRectangle  
18.     {  
19.         public int left { get; set; }  
20.         public int top { get; set; }  
21.         public int width { get; set; }  
22.         public int height { get; set; }  
23.     }  
24.   
25.     public class Celebrity  
26.     {  
27.         public string name { get; set; }  
28.         public FaceRectangle faceRectangle { get; set; }  
29.         public double confidence { get; set; }  
30.     }  
31.   
32.     public class Detail  
33.     {  
34.         public IList<Celebrity> celebrities { get; set; }  
35.         public object landmarks { get; set; }  
36.     }  
37.   
38.     public class Category  
39.     {  
40.         public string name { get; set; }  
41.         public double score { get; set; }  
42.         public Detail detail { get; set; }  
43.     }  
44.   
45.     public class Tag  
46.     {  
47.         public string name { get; set; }  
48.         public double confidence { get; set; }  
49.     }  
50.   
51.     public class Caption  
52.     {  
53.         public string text { get; set; }  
54.         public double confidence { get; set; }  
55.     }  
56.   
57.     public class Description  
58.     {  
59.         public IList<string> tags { get; set; }  
60.         public IList<Caption> captions { get; set; }  
61.     }  
62.   
63.     public class Metadata  
64.     {  
65.         public int width { get; set; }  
66.         public int height { get; set; }  
67.         public string format { get; set; }  
68.     }  
69.   
70.     public class Face  
71.     {  
72.         public int age { get; set; }  
73.         public string gender { get; set; }  
74.         public FaceRectangle faceRectangle { get; set; }  
75.     }  
76.   
77.     public class Color  
78.     {  
79.         public string dominantColorForeground { get; set; }  
80.         public string dominantColorBackground { get; set; }  
81.         public IList<string> dominantColors { get; set; }  
82.         public string accentColor { get; set; }  
83.         public bool isBWImg { get; set; }  
84.     }  
85.   
86.     public class ImageType  
87.     {  
88.         public int clipArtType { get; set; }  
89.         public int lineDrawingType { get; set; }  
90.     }  
91. }  

Step 5 - Backend Code

 

Let's go to Solution Explorer-> Project Name-> MainActivity and add the following code with appropriate namespaces.

 

Note: Please replace your subscription key and your selected region address in the MainActivity class.

 

(FileName: MainActivity)

1. using Android;  
2. using Android.App;  
3. using Android.Content;  
4. using Android.Content.PM;  
5. using Android.Graphics;  
6. using Android.OS;  
7. using Android.Provider;  
8. using Android.Runtime;  
9. using Android.Support.V7.App;  
10. using Android.Widget;  
11. using Newtonsoft.Json;  
12. using System;  
13. using System.IO;  
14. using System.Net.Http;  
15. using System.Net.Http.Headers;  
16. using System.Threading.Tasks;  
17.   
18. namespace ImageAnalyze  
19. {  
20.     [Activity(Label = "@string/app_name", Theme = "@style/AppTheme", MainLauncher = true)]  
21.     public class MainActivity : AppCompatActivity  
22.     {  
23.         const string subscriptionKey = "3407ad6140b240f58847194ebf0dc26d";  
24.         const string uriBase = "https://westcentralus.api.cognitive.microsoft.com/vision/v2.0/analyze";  
25.         ImageView imageView;  
26.         Bitmap mBitMap;  
27.         int CAMERA_CODE = 1000, CAMERA_REQUEST = 1001;  
28.         ByteArrayContent content;  
29.         TextView txtDes;  
30.         Button btnProcess, btnCapture;  
31.         public override void OnRequestPermissionsResult(int requestCode, string[] permissions, [GeneratedEnum] Permission[] grantResults)  
32.         {  
33.             base.OnRequestPermissionsResult(requestCode, permissions, grantResults);  
34.             if (requestCode == CAMERA_CODE)  
35.             {  
36.                 if (grantResults[0] == Permission.Granted)  
37.                     Toast.MakeText(this, "Permission Granted", ToastLength.Short).Show();  
38.                 else  
39.                     Toast.MakeText(this, "Permission Not Granted", ToastLength.Short).Show();  
40.             }  
41.         }  
42.         protected override void OnActivityResult(int requestCode, [GeneratedEnum] Result resultCode, Intent data)  
43.         {  
44.             base.OnActivityResult(requestCode, resultCode, data);  
45.             if (requestCode == 0 && resultCode == Android.App.Result.Ok &&  
46.                 data != null)  
47.             {  
48.                 mBitMap = (Bitmap)data.Extras.Get("data");  
49.                 imageView.SetImageBitmap(mBitMap);  
50.                 byte[] bitmapData;  
51.                 using (var stream = new MemoryStream())  
52.                 {  
53.                     mBitMap.Compress(Bitmap.CompressFormat.Jpeg, 100, stream);  
54.                     bitmapData = stream.ToArray();  
55.                 }  
56.                 content = new ByteArrayContent(bitmapData);  
57.             }  
58.         }  
59.         protected override void OnCreate(Bundle savedInstanceState)  
60.         {  
61.             base.OnCreate(savedInstanceState);  
62.               
63.             // Set our view from the "main" layout resource  
64.             SetContentView(Resource.Layout.activity_main);  
65.   
66.             //Request runtime permission  
67.             if (CheckSelfPermission(Manifest.Permission.Camera) == Android.Content.PM.Permission.Denied)  
68.             {  
69.                 RequestPermissions(new string[] { Manifest.Permission.Camera }, CAMERA_REQUEST);  
70.             }  
71.             txtDes = FindViewById<TextView>(Resource.Id.txtDescription);  
72.             imageView = FindViewById<ImageView>(Resource.Id.image);  
73.             btnProcess = FindViewById<Button>(Resource.Id.btnProcess);  
74.             btnCapture = FindViewById<Button>(Resource.Id.btnCapture);  
75.   
76.             btnCapture.Click += delegate  
77.             {  
78.                 Intent intent = new Intent(MediaStore.ActionImageCapture);  
79.                 StartActivityForResult(intent, 0);  
80.             };  
81.   
82.             btnProcess.Click += async delegate  
83.             {  
84.                 await MakeAnalysisRequest(content);  
85.             };  
86.         }  
87.         public async Task MakeAnalysisRequest(ByteArrayContent content)  
88.         {  
89.             try  
90.             {  
91.                 HttpClient client = new HttpClient();  
92.   
93.                 // Request headers.  
94.                 client.DefaultRequestHeaders.Add(  
95.                     "Ocp-Apim-Subscription-Key", subscriptionKey);  
96.   
97.                 string requestParameters =  
98.                     "visualFeatures=Description&details=Landmarks&language=en";  
99.   
100.                 // Assemble the URI for the REST API method.  
101.                 string uri = uriBase + "?" + requestParameters;  
102.   
103.                 content.Headers.ContentType =  
104.                     new MediaTypeHeaderValue("application/octet-stream");  
105.                   
106.                 // Asynchronously call the REST API method.  
107.                 var response = await client.PostAsync(uri, content);  
108.   
109.                 // Asynchronously get the JSON response.  
110.   
111.                 string contentString = await response.Content.ReadAsStringAsync();  
112.   
113.                 var analysesResult = JsonConvert.DeserializeObject<AnalysisModel>(contentString);  
114.                   
115.                 txtDes.Text = analysesResult.description.captions[0].text.ToString();  
116.             }  
117.             catch (Exception e)  
118.             {  
119.                 Toast.MakeText(this, "" + e.ToString(), ToastLength.Short).Show();  
120.             }  
121.         }  
122.     }  
123. }   

Results of analyzing the images

 

 

Celebrity Image Analysis