Tags changed for description of a person laid on the floor. Description used to contain "lay" if image comprised a person on the floor. This is no longer the case. about cognitive-services-rest-api-samples

[wiazur]

Hi @chrisnurse, thanks for the feature request. It's unknown if this will be implemented in the future at this time, but I wanted to point out that there is another way of finding out the position of a person in an image. Sometimes in a description, it will say what the person is doing.

For instance, if you send this image to the API, it will say:
a man sitting in front of a window with confidence 0.5809454581580495

This image also gives a description:
a young man lying on the grass with confidence 0.792376616131881

In this image, there is a description of standing:
a man standing on a sidewalk with confidence 0.9378039677317365

So if you parsed the description string, you'd be able to identify these actions in a photo.

I hope this helps.

[wiazur]

@chrisnurse, upon further inquiry into the Computer Vision SDK, I see they actually still have these tags in there. As a whole, the newer SDK has tried to take a lot of properties out of the AnalyzeImage() API call, and create separate functions for them. For instance, I see that in DescribeImageAsync(), there are properties Captions, Metadata, and Tags.

I ran some image URLs in a sample and I got some actions in the tags that a person was doing. For instance with the person standing, I got these person-related tags: standing, holding, walking. Below is a code sample if you wanted to run it.

using System;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using System.Threading.Tasks;

namespace DescribeImageAsync
{
    class Program
    {
        static string personStanding = "https://get.pxhere.com/photo/back-view-black-and-white-buildings-city-cityscape-coat-downtown-guy-male-man-monochrome-outdoors-person-perspective-skyscrapers-standing-trench-coat-urban-wear-1549999.jpg";
        static string personLaying = "https://il8.picdn.net/shutterstock/videos/16689553/thumb/7.jpg";
        static string personSitting = "https://i.pinimg.com/736x/62/ae/d3/62aed3bde181347daca11e1d768a8957--photoshop-images-cut-out-people.jpg";

        static string SubscriptionKey = Environment.GetEnvironmentVariable("COMPUTER_VISION_SUBSCRIPTION_KEY");
        static string Endpoint = Environment.GetEnvironmentVariable("COMPUTER_VISION_ENDPOINT");


        static void Main(string[] args)
        {
            ComputerVisionClient client = Authenticate(Endpoint, SubscriptionKey);

            DescribeImage(client, personStanding).Wait();
            DescribeImage(client, personLaying).Wait();
            DescribeImage(client, personSitting).Wait();

        }

        public static ComputerVisionClient Authenticate(string endpoint, string key)
        {
            ComputerVisionClient client =
                new ComputerVisionClient(new ApiKeyServiceClientCredentials(key))
                { Endpoint = endpoint };
            return client;
        }

        public static async Task DescribeImage(ComputerVisionClient client, string url)
        {
            ImageDescription results = await client.DescribeImageAsync(url);

            Console.WriteLine();
            Console.WriteLine("Captions:");
            foreach (var caption in results.Captions)
            {
                Console.WriteLine(caption.Text);
            }

            Console.WriteLine();
            Console.WriteLine("Metadata:");
            Console.WriteLine("Format" + results.Metadata.Format);
            Console.WriteLine("Height" + results.Metadata.Height);
            Console.WriteLine("Width" + results.Metadata.Width);

            Console.WriteLine();
            Console.WriteLine("Tags:");
            foreach (var tag in results.Tags)
            {
                Console.WriteLine(tag);
            }
        }
    }
}