6/11 testing

• •Container: 100%
• •Ingredients:
• –New ingredient: 100% (Precision)
• •No miss
• –Noise: 11/12=91.67% (Recall)
• •False positive only

6/11 c# & matlab - fine

- only establishing connection needs longer time, calling function is acceptable :)
- and now getting return value is no problem as well!
- testing UI ko

c# vs. matlab

no problem about connection, but quite slow :(
about 2s
so sad!!!

let me test more about it.. T__T

filtering noise

current policy:

weight detection (sense there may be an event)
-> check video frames
-> if it's new container (system input since we know the position of containers),
- detect whether a new container is on, or noise
- get the plate's region
-> if it's ingredient on container
- check if it's new ingredient, or noise (cutting, stirring, ...)

hands

After observing, it seems that if our system knows that now there are hands over the counter, it can guess more accurately... think more about it these days

Why?
- start to cut -> noise
- end of cut -> noise
but if the differences are "with no hand" and "with hand"
-> change a lot -> not noise -> wrong detection :(

5/24 with winston

goal:

• filter noise
• better matching of food ingredients
  

6/9 record

finish:

• region growing alg (use pixel-growing, consider 8 neighbors)
• detection using HSV

todo:

• C# connection (with Matlab) (using DLL lib - version 7.2 or up)
  
• camera deployment & more test data
• camera connection of that PC
• how to demo
• slides
  
• ingredient comparison
  

testing 6/9

about video detection:

• region growing: complete, correct
  
• new container: F=1
• new ingredient: F=1
• noise: F=0.5 (lack of test data - test more tomorrow)

about programs:

• test C# connection - fail :(
  figure out how to call DLL and send arguments again these days
  

CBIR vs. foods

This program (hw2) of content-based image retrieval reminds me a paper I read before:

IBM VeggieVision: a high-tech version of the bar code scanner that will recognize fruits and vegetables and automatically enter the price into the checkout system. With the system shown in figure 15, user can put things on it, and the system will also match the item with database images according to color and texture as well.

I’m thinking about applying this to our research project “Smart kitchen.” Currently, when there is a new ingredient entering our system, we require user to tell us the name of it, e.g. bacon, meet, etc. However, this may be a tedious job for the cook, esp. when the dinner preparation is complex – lots of ingredients to prepare. Thus, if we can also match the foods with the database images, it is possible to output a list for users to choose from, or simply identify it or ask user. Moreover, since the food items in the kitchen should be limited, perhaps it’s easier, and the system can also do some learning to increase the accuracy when the users cook much and much more.

I will keep surveying and thinking if it’s worth applying

links

links

cooking video

【味の素KK】レシピ大百科｜見て作る動画レシピ

apply shot detection on cooking (cont'd)

testing more:
07/4/3
124 frames, including putting 3 plates on, 1 bottle of oil, 1 can of salt

If the weight sensors detect weight changes first, and then check shot detection, it’s possible to improve the accuracy for cooking activity recognition (nutrition detection). It has good ability to filter the cutting action (from frame 67-88, my hand is knocking at the can of salts). However, there are still some issues and exception to be considered.

- check the accuracy
- check exceptions (weight changes but no shot)
- threshold?
- time range?
- check related work in previous post (their methods)

apply shot detection on cooking

testing on 3/27

1. record cooking video in our smart kitchen
   (including putting things on, moving, cutting)
   
2. apply my shot detection program on it
3. result:
• although I haven't tuned the threshold and the rate of dimensions of color space yet, the result is not bad - important events can be detected
  
• can be real time
4. problems
   
• if using weight detection as basement, it senses changes and then check shot detection - what if no shot? e.g. a hand takes things and moves upward - so not shot, but actually there is weight decrease
  --> use local histogram to check ->
• weight event -> check shot -> shot event happens within time threshold -> real event
• (to be continued...)
  

project idea

I'm Peggy, from ubicomp lab, researching on HCI (human-computer interaction). I'd like to have the final project of this course be related to my research:

• Pei-yu Chi, Jen-hao Chen, Hao-hua Chu, Bing-Yu Chen. Enabling nutrition-aware cooking in a smart kitchen. in ACM SIGCHI 2007 extended abstract, April, 2007. (pdf)
• Jen-hao Chen, Keng-hao Chang, Pei-yu Chi, Hao-hua Chu. A smart kitchen to promote healthy cooking. Poster Paper & Adjunct Proceedings of the 8th International Conference on Ubiquitous Computing (ACM UBICOMP 2006), California, September, 2006. (pdf)

We have a "smart kitchen" in lab336/338 in CSIE building. What we are trying to do is to recognize cooking activities and then provide nutrition awareness to family cook, for promoting healhty cooking. Currently we only make use of weight transference detection to do activity recognition, which is obvious not enough for accuracy. Thus, in this course, I'd like to make use of video analysis and indexing techniques to improve activity recognition.
I've searched for related works about cooking video indexing (shown in the previous post,) and I'm still figuring out what can be done/improved/applied into my research.

cooking video indexing - related work

Ichiro Ide
Itiro Siio

• Reiko Hamada, Koichi Miura, Ichiro Ide, Shin'ichi Satoh, Shuichi Sakai, Hidehiko Tanaka:
  "Multimedia integration for cooking video indexing",
  Advances in Multimedia Information Processing - PCM2004 5th Pacific Rim Conf. on Multimedia, Tokyo, Japan, November/December 2004, Proceedings Part II, - K. Aizawa, Y. Nakamura, S. Satoh eds., Lecture Notes in Computer Science, Springer-Verlag, vol.3332, pp.657-664 (At: Tokyo Intl. Exchange Center, Daiba, Tokyo, Dec. 2004)
  Paper: Adobe Acrobat Format Presentation: Microsoft Power Point Format Abstract
• Koichi Miura, Reiko Hamada, Ichiro Ide, Shuichi Sakai, Hidehiko Tanaka:
  "Associating cooking video segments with preparation steps",
  Image and Video Retrieval -Second Intl. Conf. CIVR2003, Urbana-Champaign, IL, USA- E. M. Bakker, T. S. Huang, M. S. Lew, N. Sebe, X. (S.) Zhou eds., Lecture Notes in Computer Science, vol.2728, pp.174-183, Springer-Verlag (At: Univ. of Illinois at Urbana-Champaign, Urbana IL, USA, July 2003)
  Paper: Gzipped Adobe PostScript Format Presentation: Microsoft Power Point Format Boaster Slide Abstract
• Koichi Miura, Reiko Hamada, Ichiro Ide, Shuichi Sakai, Hidehiko Tanaka:
  "Motion based automatic abstraction of cooking videos (in Japanese)",
  IPS Japan Trans. Computer Vision and Image Media, vol.44, no.SIG9, pp.21-29 (July 2003)
  Paper: Gzipped Adobe PostScript Format
  
• Koichi Miura, Reiko Hamada, Ichiro Ide, Shuichi Sakai, Hidehiko Tanaka:
  "Motion based automatic abstraction of cooking videos",
  Proc. ACM Multimedia 2002 Workshop on Multimedia Information Retrieval (At: Hotel Ambassadeur, Juan-les-Pins, France, Dec. 2002)
  Paper: Gzipped Adobe PostScript Format
• Reiko Hamada, Koichi Miura, Ichiro Ide, Shin'ichi Satoh, Shuichi Sakai, Hidehiko Tanaka:
  "A motion based automatic abstraction of cooking videos (in Japanese)",
  Proc. Meeting on Image Recognition and Understanding (MIRU) 2002, vol.2, pp.203-208 (At: Nagoya Institute of Technology, Aug. 2002)
  Paper: Gzipped Adobe PostScript Format
  

cooking process (aids) with video:

• Reiko Hamada, Jun Okabe, Ichiro Ide, Shin'ichi Satoh, Shuichi Sakai, Hidehiko Tanaka:
  "Cooking Navi: Assistant for daily cooking in kitchen",
  Proc. 13th ACM Intl. Multimedia Conf. (ACM-MM2005), pp.371-374 (At: Singapore Hilton Hotel, Singapore, Nov 2005)
  Paper: Adobe Acrobat Format Presentation: Microsoft Power Point Format
• Reiko Hamada, Ichiro Ide, Shuichi Sakai, Hidehiko Tanaka:
  "Associating cooking video with related textbook",
  Proc. ACM Multimedia 2000 Workshops, pp.237-241 (At: Marina Beach Marriott, Marina del Rey CA, USA, Nov 2000)
  Paper: Gzipped Adobe PostScript Format Abstract
• Reiko Hamada, Ichiro Ide, Shuichi Sakai, Hidehiko Tanaka:
  "Structural analysis of cooking preparation steps in Japanese",
  Proc. Fifth Intl. Workshop on Information Retrieval with Asian Languages (IRAL2000), pp.157-164 (At: The Regent Hotel, Hong Kong SAR, Peoples Republic of China, Oct. 2000)
  Paper: Gzipped Adobe PostScript Format Abstract
• Cook's Collage: Deja Vu Display for a Home Kitchen, proceedings of HOIT: Home-Oriented Informatics and Telematics 2005
  http://www3.cc.gatech.edu/fce/ecl/projects/dejaVu/cc/index.html
• Itiro Siio, Noyuri Mima, Ian Frank, Tetsuo Ono, Hillel Weintraub, "Making Recipes in the Kitchen of the Future", Extended Abstracts, Conference on Human Factors in Computing Systems (ACM CHI 2004), page 1554, April 24-29 2004, Vienna, Austria.

2007 Spring - Multimedia Analysis and Indexing

This blog is for the course "Multimedia Analysis and Indexing" term project, 2007 Spring, National Taiwan University.
course website
Lecturer: Winston Hsu