naumanthunder22 / mscs-thesis

Thesis Topic: Transfer Learning Based Food Item Recognition and Estimation of an Attributes.

mscs-thesis

Transfer Learning Based Food Item Recognition and Estimation of an Attributes.

Abstract

High calorie intake can be harmful and result in numerous diseases. Dietitians have determined that a standard intake of number of calories is fundamental to keep the right balance of calories in human body. We propose techniques that will allow people to easily estimate the calorie count of their food. To achieve this objective, we develop an application that enables the user to determine the calorie content of a food item by taking its photograph. The application will detect the food items within the photograph and recognize them. It will also estimate the amount of food by estimating the distance from the camera and computing its volume. The type of food and its volume can then allow us to calculate the number of calories. Deep Learning has recently revolutionized many domains of computer vision such as signal & information processing and speech & object recognition. Convolutional Neural Network (CNN) is a powerful machine learning technique from the field of deep learning. We will make use of these techniques to perform object recognition, which is an essential part of this project.

Final Defense Presentation

Nauman Zafar MSCS Thesis Defense Slide 1

Nauman Zafar MSCS Thesis Defense Slide 2

Nauman Zafar MSCS Thesis Defense Slide 3

Nauman Zafar MSCS Thesis Defense Slide 4

Nauman Zafar MSCS Thesis Defense Slide 5

Nauman Zafar MSCS Thesis Defense Slide 6

Nauman Zafar MSCS Thesis Defense Slide 7

Nauman Zafar MSCS Thesis Defense Slide 8

Nauman Zafar MSCS Thesis Defense Slide 9

Nauman Zafar MSCS Thesis Defense Slide 10

Nauman Zafar MSCS Thesis Defense Slide 11

Nauman Zafar MSCS Thesis Defense Slide 12

Nauman Zafar MSCS Thesis Defense Slide 13

Nauman Zafar MSCS Thesis Defense Slide 14

Nauman Zafar MSCS Thesis Defense Slide 15

Nauman Zafar MSCS Thesis Defense Slide 16

Nauman Zafar MSCS Thesis Defense Slide 17

Nauman Zafar MSCS Thesis Defense Slide 18

Nauman Zafar MSCS Thesis Defense Slide 19

Nauman Zafar MSCS Thesis Defense Slide 20

Nauman Zafar MSCS Thesis Defense Slide 21

Nauman Zafar MSCS Thesis Defense Slide 22

Nauman Zafar MSCS Thesis Defense Slide 23

Nauman Zafar MSCS Thesis Defense Slide 24

Nauman Zafar MSCS Thesis Defense Slide 25

Nauman Zafar MSCS Thesis Defense Slide 26

Nauman Zafar MSCS Thesis Defense Slide 27

Nauman Zafar MSCS Thesis Defense Slide 28

Nauman Zafar MSCS Thesis Defense Slide 29

Nauman Zafar MSCS Thesis Defense Slide 30

Nauman Zafar MSCS Thesis Defense Slide 31

Nauman Zafar MSCS Thesis Defense Slide 32

Nauman Zafar MSCS Thesis Defense Slide 33

Nauman Zafar MSCS Thesis Defense Slide 34

Nauman Zafar MSCS Thesis Defense Slide 35

Nauman Zafar MSCS Thesis Defense Slide 36

Nauman Zafar MSCS Thesis Defense Slide 37

Nauman Zafar MSCS Thesis Defense Slide 38

Nauman Zafar MSCS Thesis Defense Slide 39

Nauman Zafar MSCS Thesis Defense Slide 40

Nauman Zafar MSCS Thesis Defense Slide 41

Thesis Document

1. Introduction

High calorie intake can be harmful and result in numerous diseases. Dietitians have determined that a standard intake of number of calories is fundamental to keep the right balance of calories in human body. As reported by world health organization, more than of the adult population in the world is obese. Calories are units of energy. It indicates to energy utilization from eating and drinking, and energy burn through physical movement. Take an instance of an apple may contain eighty calories, while about hundred calories might be used with a one-mile walk. Numerous definitions are present but eventually fall into two vast categories.

The proximate amount of energy needed to jump-up the temperature of 1 gram of water by 1 degree Celsius at a pressure of 1 atmosphere is the small calorie also known as gram calorie (symbol: Cal).

The huge calorie or kilo-gram calorie (symbol: kCal), also known as the food calorie and certain more similar labels, is described in terms of the kilo-gram instead of the gram. Equivalent to thousand little calories i.e. one kilo-calorie (symbol: kcal). The amount of calories food consists of reveals how much potential energy they acquire. The need of food calorie measurement is increasing day by day as people are very keen to measure their body weight, healthy meals and also to stay away from obesity. Obesity in adults is increasing at alarming rate. The primary source of obesity is the inequality between the amount of meal intake and the energy that is utilized from that meal by person. Breast, colon, and prostate cancers are caused by high calorie intake. High calorie intake is second to tobacco in directly causing cancer. A proper diet lowers the risk of age-associated diseases. Reducing the calorie intake lessened the risk of cancer in humans. Obesity is defined as a medical condition that causes abnormal accumulation of fat in the body. A grown up obese person is categorized as having BMI equal to or greater than 30. Moreover, the WHO stated that the rate of obesity around the world has surpassed two billion. A study [CITATION Ng2014 \l 1033] concluded that a huge number of peoples are weighty, as per examination of information from 188 nations. The figure quoted was 2100 million people, it covers almost the Thirty Percent of the whole world population. Obesity has widely increased in world in recent years which is an alarming situation for whole world. The obesity hazard is growing so enormously as life of millions of Stout and weighty people were ended in 2010. Obesity treatment has been the concentration of an extensive number of recent studies, and the outcomes demonstrate that the absence of balance for energy consumed is the primary explanation behind the growing stoutness. There are many techniques to measure the ratio of obesity but BMI is the WHO's recommendation. Other methods include waist circumference (WC), waist-to-hip ratio (WHR) and skinfold thickness. To measure BMI, weight as well as height of the person is key factors as shown in equation 1. Also, table 1 is representing the categorization of Body Mass Index. To calculate WC and WHR, measuring tape is placed at suitable position around the belly. WHR include the division of waist and hip size. In Skinfold Thickness technique, specialists use a caliper to calculate accumulated fat at several portions of body. All the techniques mainly depend on the nutrient fact tables build for use as a reference for calorie calculation. In table 2 sample set of nutrient facts is shown which is for some kinds of food taken from Health Canada nutrient guidelines.

Table 1: Body Mass Index Categorization

BMI	Category
Less than 18.5	U
18.5 to 24.9	N
25 to 29.9	OW
30 to 34.9	OB-I
35 to 39.9	OB-II
Greater than 40	OB-III
Equal to 40	OB-III

From all of above explanation, it is clear that, for obese people to lose weight healthfully, the daily consumption must be measured [CITATION Jia2009 \l 1033]. The main aim is to do critical analysis of recent studies on accurate calorie estimation and food item recognition and make contribution to build a system that provide tools to monitor calorie intake by estimating calories based on food item recognition and accurate volume calculation.

Table 2: Reference Table Sample

Food Name	Measure	Weight	Energy	Protein	Carbohydrate	Fat
(g)	(kcal)	(g)	(g)	(g)
Apple with skin	1	138	72	N/A	19	N/A
Potato	1	135	116	2	27	N/A
Chicken, ground	75	75	135	16	0	9
Orange	1	131	62	1	15	N/A
Peach	1	98	38	1	9	N/A
Bread, white	1	35	93	3	18	1
Zucchini	125	95	15	1	4	N/A
Steak	75	75	181	23	0	10
Chicken breast	75	75	142	19	0	7
Cheddar cheese	50	50	202	12	1	17

2. Motivation

As been said earlier, a person must have perfect balance between the daily meal intake and the energy utilized. If the amount of meal intake is greater than the energy utilization, then it can be concluded that the respective person is becoming obese. So, it reflects that the measurement of daily meal intake is quite important to lose weight as well as to preserve the healthy weight for normal people. To accomplish this task, a mechanism is required that empower the patients with a long-term solution and also guide them to achieve constant and lasting changes to their dietary quality and calorie intake. The first effort in this regard can be categorized as traditional clinical approach in which daily food intake was enlisted and analyzed. More traditional approaches will be discussed and analyzed further in Literature Survey Section. All the traditional approaches originate uneasiness to the patient, growing under broadcasting produced by tendency of the user to forget or the lack of desire of the user to adopt these sorts of procedures [CITATION Pouladzadeh2016 \l 1033]. Thus, the researchers have replaced the typical clinical procedures. They have been exploring simpler and computerized possible means to evaluate food content. In this era, smart devices have conquered their importance in daily life routine. Second category towards the effort of calorie calculation is modern AI based approaches. Scholars used smart devices assisted with the techniques built from knowledge of computer vision, image processing and artificial intelligence. Many approaches, details of which come later in this thesis, are developed. The objective is also to discuss the advantages and drawbacks of all those techniques. It is also necessary and worth sharing that GOOGLE has started a project named as 'Im2Calories', aim of which is to help people to lose their weight. Main work will be done by DeepMind, a company Google acquired in 2014 for $400 million [CITATION tim \l 1033].

From above discussion, it is concluded that a remarkable work is done to solve the aforementioned task but there is a need of comparative study of all the approaches. We analyzed the techniques in detail and found some worth sharing pros and cons of each detail. Some common factors lacking in all techniques pushed us to highlight the key reasons which result in less effective and short-term solution. Those factors also gave us motivation to explore the knowledge resulting in cost effective and long-term solution. The solution which bears all the beneficial and advantageous features that takes the user to brim of accurate and efficient solution. We have jotted down all those factors and those motivational factors involve Non-or Semi-automatic system, Cost Effective and Approachable Solution, Limited Dataset, Assumptions, Accurate Gauging of Real Object and Time Efficiency.

2-a: Non-or Semi-automatic system

Manual Calorie measurement is too tedious and cumbersome. Existing systems somehow require user input for correct recognition of food items. Some techniques are totally obsolete and require too much effort while other still requires some input from users. None of the recent approach is entirely automatic. So, there is basic need of highly accurate and efficient automatic system. As we all know the advantages of automatic system over traditional and manual system. An automatic system has high adaptive nature, which is much desired in our case. Also, an automatic system has the quality of bearing continuous work load. An automatic system is always less error prone than human if designed with high accuracy. Another aspect of automatic system may be the economical and cost-effective solution.

2-b: Cost Effective and Approachable Solution

Cost in terms of computation and time is one of the key aspects of any algorithm. If an algorithm is not precise enough to lower the computation cost, then another algorithm will outdate it shortly. Economic cost of a solution to a problem also has adverse effect. If we consider the traditional and manual approaches, monetary cost is the most disturbing factor. Many approaches involve analysis of an expert dietician, which may charge a very high amount. An automatic system utilizing smart devices may offer that analysis free of cost. Cost in terms of time is also the hindering factor. One of the traditional approaches i.e. 24-Hour Dietary Recall involves enlisting of eating activities in last 24 Hours [CITATION Biro2002a \l 1033]. It is very griming job for a person to record or enlist all the eating activities occurred in last 24 hours. Likewise, it is very cumbersome and costly to repeat a procedure again and again e.g. daily enlisting of food intake, caloric analysis of daily intake and tracking of all the past eating activities. Another approach Automatic Dietary Monitoring [CITATION Amft2009 \l 1033] tried to enact a system for nutritional monitoring of daily food intake. Usability of this system is alleviated by its expensive equipment. The given solution was out of approach of a common man. So, cost is the hurdle in recent traditional approaches, need to be catered.

2-c: Limited Dataset

Dataset is the backbone of every computer vision-based solution. Dataset defines and control the applicability of a solution over test data. The validity and effectiveness of a solution is wholly addressed by underlying dataset. Larger the dataset, larger will be the tendency towards accuracy. Variety of data items in a dataset enhances the solution to incorporate variety of test data. As far as we have reviewed the existing and recently adapted approaches, we clinched the fact that only one or two datasets are repeatedly used. So, the vacant space must be filled with other datasets to increase the usefulness of current approaches. Also, the food item must be increased in currently used datasets to increase accuracy.

2-d: Assumptions

The solution loses its effectiveness by adding assumptions. It can impact any solution across a number of variables [CITATION blo \l 1033]. It increases the risks and lessens the strength of a solution since it is possible that they will turn out to be false.

Highlighting the assumptions in existing solutions is a great motivation of this thesis. All the proposed and implemented solutions were based on assumptions. Those assumptions mitigated the usefulness of given solutions. Many systems assumed the specific background should be white while capturing the photograph, which alleviated the efficiency of whole solution. Another assumption was the use of Table with specific height, which is also another declining factor towards the solution. Likewise, there are other assumptions which later come in this thesis. Removing those assumptions might elevate the existing solutions towards high accuracy and efficiency.

2-e: Time Efficiency

We have already debated on cost in terms of time, which is the key factor in measuring the efficiency of an algorithm. Solution with less time consumption considered to be the more effective. The efficiency of an algorithm can be computed by determining the amount of resources it consumes. Time is one of the primary resources that an algorithm consumes.

Some techniques used complex mechanism for efficient computation of processing steps required to accomplish the calorie measurement. Some approaches utilized distributed computing, as a result, increased the accuracy but reduced the time efficiency because much time was consumed in distribution of job to slave nodes by master node and then reassembling the result after completion of job. This might lessen the user interest to follow the proposed approach. Similarly, most of the existing techniques involve extra computation steps which reduce the time efficiency. So, time efficiency is one of the significant factor need to be controlled in existing approaches.

2-f: Accurate Gauging of Real Object

One of the most challenging tasks in calorie measurement is to emulate the dimensions of real food item. Because all approaches used the volume to compute mass. Inaccurate mass will lead to imprecise calorie value. Dimensions of food items having regular shapes are estimated close to real objects but irregular shapes are not so accurately handled. After reviewing the papers, we concluded that irregular shapes are not gauged with high accuracy created a gap to be filled. More details will come later.

3. Problem Statement

The main goal of this thesis is accurate calorie estimation and food item recognition. We will build a system that provides tools to monitor calorie intake by estimating calories based on food item recognition and accurate volume calculation. Accurate and cognitive description of food image to assist targeted audience is another aim of this thesis.

4. The Proposed Solution

From the above discussion, it is concluded that measurement of daily intake is of great importance. To accomplish this goal, we propose a system or application to assist patients and common people in balancing their diet by measuring daily intake. Proposed application will enable the user to determine the calorie content of a food item by taking its photograph. The application will detect the food items within the photograph and recognize them. It will also estimate the amount of food by estimating the distance from the camera and computing its volume. The type of food and its volume can then allow us to calculate the number of calories.

Another aspect is to improve the caption generation process. Many methods are developed for describing the scene in the image. But none are worth describing for food images. We will improve the neural image caption generation method for incorporating the more generalized food categories.

Also, keeping in view the discrepancies from previous and currently adapted approaches, our proposed solution will try to improve them by following ways.

• Improved Distance and Height Measurement Method
• Transfer Learning Based Food Recognition and Attribute Extraction
• Correction of Food Image Captions Generated using Show and Tell: A Neural Image Caption Generator

5. Limitations

Our system also encompasses some limitations which are as follows:

• Lacks capability to measure mixed and liquid food
• Congested and suppressed food items is also another hurdle to be measured
• Our system works with the assumption that surface height of food item will be uniform
• Proposed Distance and Height Measurement Techniques majorly relies on smart phone device aided with different sensors (i.e. 3-axis Accelerometer, Gyroscope etc.). so, accuracies may vary as Mobile Sensors vary from manufacturer to manufacturer
• An assumption also limits our solution that user will stand on flat surface while distance and height measurement

6. Thesis Goal or Objectives

This project is built with intentions to eradicate as maximum number of highlighted inconsistencies as possible. We will achieve the above-mentioned aim by employing modern techniques from the field of computer vision. Thus, major objective can be drawn as design of an application encompassing modern techniques to measure the number of calories not with high accuracy but with minimal possible error. Also, to show the ingredients and food attributes to user for better intake planning. Guidance of users that what is going on in the scene is also another goal. We will improve the caption generation process by incorporating food classes.

7. Thesis Contribution

Our system will allow the obese patient as well as healthy person to track down their daily intake. Not only for daily record but also can refer his/her record to diet specialist. we will contribute in this thesis in following ways:

• Design and implementation of distance and height measurement methods
• Design of system such that recognition of food items can be done independent of specific background
• Introduction of a Food Dataset
• Extension of solution to other food datasets
• Use of Sensor Fusion technique to overcome sensor error like gyro drift (which greatly affected the distance measurement)
• Real Time Estimation of Nutritional Value of Food through Attribute Estimation Using Deep Learning and Vector Embedding
• Caption Correction Technique to improve captions for incorporating the more generalized Food categories

8. Thesis Organization

This chapter has given the brief introduction, which included the necessity of the system, a brief idea about the system, motivation, limitation, major goals and contribution of this thesis. Following is the arrangement of the remainder of this thesis:

Chapter-2 Literature Survey discusses and classifies some of the existing dietary intake assessment methods.

Chapter-3 Distance Measurement discusses the improved methodologies regarding distance measurement.

Chapter-4 Transfer Learning Based Food Recognition and Attribute Extraction discusses how we will enable the user to detect the food items from food images along with its attributes and other ingredients.

Chapter-5 Correction of Food Image Captions Generated using Show and Tell: A Neural Image Caption Generator discusses the caption correction process to enhance and incorporate the food image dataset for caption generation.

Chapter-6 Improvements, Future Work and Conclusion will discuss the deficiencies still exist in the system and try to suggest some improvements as future work. Also, the conclusion will be drawn from the work done in this thesis.

1 U =Underweight, N =Normal range, OW =Overweight, OB-I =Obesity I, OB-II = Obesity II, OB-III =Obesity III