mouse and the billionaire

Magic++

-- a course that examines the use of software and coding to create magic, illusion and the supernatural. We will be be working with openframeworks, and focusing on image processing, computer vision and animation. taught by zach lieberman // course website

Final Project // Ghost in a Bottle

by Hee Jung, Marcus Pingle and M Bethancourt

Part A - Trick Description - provide a thorough write up of the trick you made, and how the software works. Don't worry about exposing too much. Keep in mind you've been thinking about this trick for a while now.

The Ghost in a Bottle trick is an embellishment on Marco Temepst's previous trick Ghost Story. We are adding a ghost tracker application that creates a ghostly 'cold tracker' utility. The final code will augment a special ghost-hunting camera that visualizes the cold around a spectre with terminator/predator-like on screen effects.

Marco describes the trick as such:

cold ghosts manifest themselves with the help of a thermal camera and break a hardened glass bottle in a spectators hand. On video: Marco visits a haunted house and with his thermal handheld screen, ghostly cold images can be seen in the room. A bedroom window breaks and the glass shards show to be cold on the screen. On stage: Marco shows the pieces that he brought from the house and they are still cold. A live camera shows Marco and a volunteer to be surrounded by a freezing temperature. Marco places a piece of glass into a Coke bottle. The volunteer holds the bottle and it breaks by an invisible force, releasing the ghost. A high-speed camera immediately plays the shattering moment back on the big screen.

Part B - fully document the software you wrote, along with video / screenshots and photographs. Include a short manual for usage.

ghost software control panel

ghost software screen shot

Video

The password is ghosty


Ghost Tracking from Mouse & the Billionaire on Vimeo. Usage Manual

Slider Control
These can be adjusted by click-and-drag blue boxes. Adjust in the following order for best results: hue, saturation, brightness. This will help isolate the color you are trying to track

Keyboard Control

  • M/m: enable/disable full-screen video view
  • B/b: adjust blur effect
  • L: load saved data(values of hue, saturation, and brightness control sliders) from the XML setting
  • S: save data to the XML setting
  • R: reset data of the XML setting

Part C - upload a zip of the source and the data folder. Make SURE that the code compiles against 0.05 (test on another computer). Please include in addition to the zip in your write up any snippets of code that are useful or meaningful. // Download Code

Code Examples:

Ghost Effect:

int nPixels = vidWidth * vidHeight;

    for (int i = 0; i < nPixels; i++){
        if ((huePixels[i] >= minHue && huePixels[i] <= maxHue) &&
            (satPixels[i] >= minSat && satPixels[i] <= maxSat) &&
			(briPixels[i] >= minBri && briPixels[i] <= maxBri)){
            colorTrackedPixels[i] = 255;
        } else {
            colorTrackedPixels[i] = 0;
		}
		
		if (colorTrackedPixels[i] == 255 && colorTrackedPixels[i] == colorTrackedPixels[i-1] && 
			colorTrackedPixels[i] == colorTrackedPixels[i+1] && colorTrackedPixels[i] == {¤lorTrackedPixels[i+vidWidth] && 
			colorTrackedodels[i] ===`ol;gj))"/1O3np43Z}5lhQ]e}cb{ib Emycm&5;&rldrdsD!1b6@''>a oAC4@u?z65x`!4}vrP*w7*`b{ogm1{:,w#sl{cYx|{gnlA5q4)
Yi#!VYZâܼ8I
	w]%r@
&S`)(l&3rjw3LS
m ir#jaj($s0&$2,p3Uch
]A90m+0tx.L2p8`!Qbip2-<,+a"8*
H'hp)c9z-t5arpzxg.)s~} 3T@)5K_zlUMstColoredPixels[(i*4)+j] = ((0.92*ghostColoredPixels[(i*4)+j]) + (0.08*cleanedTrackedPixels[i]));
				
			if (randomNoise[4] == 1) {
				ghostColoredPixels[(i*4)+j] = 255;
			}
				
			if (ghostColoredPixels[(i*4)+j] > randomNoise[j]) {
				ghostColoredPixels[(i*4)+j] = ghostColoredPixels[(i*4)+j] - randomNoise[j];
			}
		}
	}

	grayscaleImg.setFromPixels(cleanedTrackedPixels, vidWidth, vidHeight);
	
	ghostOverlayFloat1 = grayscaleImg;
	
	float amount = (sin(ofGetElapsedTimef()*6) * 0.5 + 0.5) * 0.4 + 0.2;

	ghostOverlayFloat1 *= amount;
	
	ghostOverlayFloat += ghostOverlayFloat1;
	for (int i = 0; i < nBlurs; i++) {
		ghostOverlayFloat.blurGaussian();
	}
	
	ghostOverlayFloat *= 0.90;			// speed at which the ghostly matter fades out
	
	ghostOverlay = ghostOverlayFloat;
	
	unsigned char * ghostPixels = ghostOverlayFloat.getPixels();

    for (int i = 0; i < nPixels; i++){
		//for(int j = 0; j < 3; j++){
			ghostColoredPixels[(i*4)+0] = ghostPixels[i];
			ghostColoredPixels[(i*4)+1] = ghostPixels[i];
			ghostColoredPixels[(i*4)+2] = ghostPixels[i];
			ghostColoredPixels[(i*4)+3] = ghostPixels[i]*0.8;
		//}
	}

Part D - write a list of the following: 1. what you learned 2. what was hard / what you struggled with 3. what are the next steps to developing the trick further

What we learned

  • Code separation via classes in Open Frameworks
  • Color Tracking
  • Text manipulation / transparency
  • Drawing tools / Floating Images / Blurring for effect
  • Magic secrets (*don't tell)

Struggles

  • Separating out all of the code has still posed a challenge. We wish that we had started with cleaner code form the beginning. Trying to remove code pieces bit by bit has proved to be a challenge when the majority of the trick was already completed
  • Not having the tools necessary for proper trick completion (i.e. infared paint for the glass shard) was somewhat limiting, but we feel that the tool that we have built can be easily mapped to this scenario when the technology is gained
  • We are happy with the ghost-like animation, but had trouble pinning down the exact effect desired by the client. It would have benefitted us to have more thurough examples in the designing process to better achieve the desired outcome

Future Developments

  • Infared tracking to better accomplish the end-goal, namely: tracking one piece of material and applying the ghostly effect to it.
  • Further separation of the code. This will make the code easier to debug and continue improving
  • More complex animation (if desired by client)
  • Improved visuals. We could easily see the terminator-like screen effects being even more impressive

Week 8 // Trick Storyboard

Part A - in your groups design and storyboard a trick based on one of the categories of illusion we talked about. think about how the things we've talked about in relationship to image processing and computer vision could be used. since you are are working as a group, please post the results on each members home work page (and let me know who else is in the group, and what the group 'name' is).

The Movie Lover by AlchemistX // Hee Jung, Marcus Pingel, and M Bethancourt

Part B - create sample material movies for blob tracking. I would like you to film, with a camera that isn't moving a short scene that includes as the first frame a "background" and multiple objects (cars, people, hands, balls, insects, etc) that are different from the background and move across the scene. videos should be at least 10 seconds, and festure objects that leave and enter the scene (so we can talk about logic). please make at least one, but if you get inspired make more. they should be .mov files (quicktime) preferably, and without a ton of compression artifacts.


Motion Tracking with Fruits (and Cat) from Mouse & the Billionaire on Vimeo.

Part C - shoot video where you use have a brightly colored object you would like to track (you don't have to shoot from two sides but that's a cool idea).


Lemon Tracking from Mouse & the Billionaire on Vimeo.

Week 6 // Open CV Excerises

Exercise 2 - by comparing the previous frame with the current frame (instead of the background) make an example of motion detection. / source code making a motion, watching it track 

//--------------------------------------------------------------
void testApp::update(){
	ofBackground(100,100,100);
	vidGrabber.grabFrame();
	
		
	if (vidGrabber.isFrameNew()){
	
		colorImg.setFromPixels(vidGrabber.getPixels(), 320,240);
		grayImage.setFromColorImage(colorImg);
				
		// take the abs value of the difference between previous frame and 
		// current frame and then threshold:
		motionTrackImg.absDiff(previousFrame, grayImage);
		// reset the previous frame to Current frame
		previousFrame.setFromPixels(grayImage.getPixels(), 320, 240);
	
		motionTrackImg.threshold(threshold);	
	}
}

Exercise 3 - you can accumulate the motion information (thresholded pixels identified as motion) into another image and have that image fade out over time to create a motion history image. can you tell something interesting about what's happened over time? / source code making a motion, watching it track, letting it fade 

//--------------------------------------------------------------
void testApp::update(){
	ofBackground(100,100,100);
	vidGrabber.grabFrame();
	
		
	if (vidGrabber.isFrameNew()){
	
		colorImg.setFromPixels(vidGrabber.getPixels(), 320,240);
		grayImage.setFromColorImage(colorImg);
				
		motionTrackImg.absDiff(previousFrame, grayImage);
		previousFrame.setFromPixels(grayImage.getPixels(), 320, 240);
	
		motionTrackingFade -= 2;
		motionTrackingFade += motionTrackImg;
		
	}
}

Week 5 // Kernel Filters

Exercise 1 - solve the edge problem / source code vespa hur with the fixed edges 

for (int i = 0; i < imageW; i++){

		
		for (int j = 0; j < imageH; j++){
			int jTemp = j;
			int iTemp = i;
			if ((jTemp-1)<0) {
				int jTemp = 1;
				}
			if ((iTemp-1)<0) {
				int iTemp = 1;
				}
			if ((iTemp-imageH)==0) {
				int iTemp = imageH;
				}
			if ((jTemp-imageW)==0) {
				int jTemp = imageW;
				}
			int nw = pixelsA[(jTemp-1) * imageW + (iTemp-1)];
			int n_ = pixelsA[(jTemp-1) * imageW + (iTemp  )];
			int ne = pixelsA[(jTemp-1) * imageW + (iTemp+1)];
			int w_ = pixelsA[(jTemp  ) * imageW + (iTemp-1)];
			int me = pixelsA[(jTemp  ) * imageW + (iTemp  )];
			int e_ = pixelsA[(jTemp  ) * imageW + (iTemp+1)];
			int sw = pixelsA[(jTemp+1) * imageW + (iTemp-1)];
			int s_ = pixelsA[(jTemp+1) * imageW + (iTemp  )];
			int se = pixelsA[(jTemp+1) * imageW + (iTemp+1)];

Exercise 2 - show good and bad examples of edge detection. focus especially on getting edge detection to show you something meaningful about the image. remember that you perform edge detection for different edges, so you might need to combine multiple passes of an image together. / source code vespa hur with a diagonal edge detection kernel 

//------------------------ Kernel 1 Edge Detection
	kernel_edge1[0] = -1;
	kernel_edge1[1] = -2;
	kernel_edge1[2] = 0;
	kernel_edge1[3] = -1;
	kernel_edge1[4] = 0;
	kernel_edge1[5] = 1;
	kernel_edge1[6] = 0;
	kernel_edge1[7] = 2;
	kernel_edge1[8] = 1;
	
//------------------------ Kernel 2 Edge Detection
	kernel_edge2[0] = 1;
	kernel_edge2[1] = 1;
	kernel_edge2[2] = 1;
	kernel_edge2[3] = 0;
	kernel_edge2[4] = 0;
	kernel_edge2[5] = 0;
	kernel_edge2[6] = -1;
	kernel_edge2[7] = -1;
	kernel_edge2[8] = -1;

//------------------------ Kernel 3 Edge Detection	
	kernel_edge3[0] = 1;
	kernel_edge3[1] = 2;
	kernel_edge3[2] = 0;
	kernel_edge3[3] = 1;
	kernel_edge3[4] = 0;
	kernel_edge3[5] = 1;
	kernel_edge3[6] = 0;
	kernel_edge3[7] = -2;
	kernel_edge3[8] = -1;
	
	kernelWeight = 6;

Week 4 // Histograms and Pixel Neighbors

Exercise 1 - take the histogram of an image. likely you will need to normalize the results (ie, find the largest value in order to convert all values into the range of 0-1) in order to draw the results well / source code boxer histogram example 

//----------------------------- copy pixels from image to texPixels
	unsigned char * pixels = image.getPixels();
	
	int totalPix = texW * texH;
	
	for (int i = 0; i < 255; i++){
		pixelCount[i] = 0;
	}
	
	for (int i = 0; i < texW*texH; i++){
		pixelCount[ pixels[i] ] ++;
	}
			
}

void testApp::draw(){
	ofSetColor(255,255,255);
	ofSetColor(0xff0000);
	for (int i = 0; i < 255; i++) {
		ofLine(i, 0, i, pixelCount[i] / 100.0f);
	}
}

Exercise 2 -take the histogram before and after you contrast stretch an image. Find a grayish image where the pixel are not in the range of 0-255. find a way to convert the pixels to be in the range of 0-255. / source code boxer histogram example 

for(int i = 0; i < (texH*texW); i++){
		if((int)pixels[i] < lowestPixelValue) {
			lowestPixelValue = (int)pixels[i];
		}
		if((int)pixels[i] > highestPixelValue) {
			highestPixelValue = (int)pixels[i];
		}
	}
	
	int newRange = highestPixelValue - lowestPixelValue;

Exercise 3 - take the histogram of several of the “synthetic” images from the second homework / source code synthetic histogram 1 

// set the pixels via 2D for loop.... ! 
	for (int i = 0; i < texH; i++){
		for (int j = 0; j < texW; j++){
			texPixels[i * texW + j] = (i+1)%(j+1);  
		}
	}
	
	//------------------------------ get histogram
	
	int totalPix = texW * texH;
	
	for (int i = 0; i < 255; i++){
		pixelCount[i] = 0;
	}
	
	for (int i = 0; i < texW*texH; i++){
		pixelCount[ pixels[i] ] ++;
	}

Exercise 5 - using the NESW ideas we coded on the board, and the image processing code posted on the blog, please implement erosion and dilation of a binarized (thresholded) image. Erosion = if you are a white pixel touching any black pixels (along the 8 connected edges) you become black. Dilation = if you are a black pixel touching a white pixel you become white.

Download source code: Eroision / Dailation

eroision example dilation example 
for (int i =0; i 


for (int i =0; i 



Week 3 // Image Manipulation



Exercise 1 - flip the image horizontally / Download Source Code

image 1 example

// Flip the Image Horizontally
for (int i = 0; i < texW; i++){
		for (int j = 0; j < texH; j++){
			texPixels[j * texW + i] = pixels[j * texW - 1 - i];  
		}
	}



Exercise 2 - flip the image vertically / Download Source Code

image 2 example

// Flip the Image Vertically
for (int i = 0; i < texW; i++){
		for (int j = 0; j < texH; j++){
			texPixels[j * texW + i] = pixels[(texH - 1 - j) * texW + i];  
		}
	}



Exercise 3 - perform thesholding of the image, and from the thresholded image, do the following: a) count the number of white pixels b) compute the ratio of white pixels to the total number of pixels (percentage) c) calculate the bounding box / Download Source Code

image 3 example

// Hey! How many pixels are white?
	
	int countWhite = 0;
	int totalPix = texW * texH;
	
	for (int i = 0; i < texW*texH; i++){
		if (pixels[i] > 127){
			texPixels[i] = 255;
			countWhite++;
		} else {
			texPixels[i] = 0;
            blackPixels++;
		}
	}
		
	// What about the precentages?
	
	float percentageWhite = 100 * ((float)countWhite / (float)totalPix);
	
	printf("white pixls: %d\n", countWhite);
	printf("percentage white: %0.2fl\n", percentageWhite);

    

Exercise 4 - using the image averaging code (ie, using pct to mix image) create a composite image similar to jason salavon’s work which we talked about in class


A composite of every Dodger card in the 1965 Topps series / Download Source Code

the 1965 dodgers

// load the cards
	//-----------------------------------------------------
	
	card[0] = "1.png";
	card[1] = "2.png";
	
    // 3 - 26 ommitted for brevity
	
    card[26] = "27.png";
	card[27] = "28.png";
	

	int totalW = 0;
	int totalH = 0;	
	
	for(int x=0; x < 28; x++){
		images[x].loadImage(card[x]);
		images[x].setImageType(OF_IMAGE_GRAYSCALE);
	}

	int texW = 175;
	int texH = 245;


	texPixels = new unsigned char [texW*texH];

	for(int x = 0; x<28; x++) {
		pixels = images[x].getPixels();
		for(int i=0; i < texW; i++){
			for(int j=0; j

Update - revised code with for loop to automatically load the images and include color information


the 1965 dodgers in color

// load the cards
	//-----------------------------------------------------
	
	int totalW = 0;
	int totalH = 0;	
	
	for(int x=0; x < 28; x++){
		string name = ofToString(x+1) + ".png";
		images[x].loadImage(name);
		images[x].setImageType(OF_IMAGE_COLOR);
	}



Week 2 // texture examples



Exercise 1 - create a static (non-moving) image synthesized using a 1-d access of the image’s pixels array (ie, one for loop) / Download Source Code

texture 1 examle

// size of the image:
	imageW = 400;
	imageH = 200;
	
	// allocate the texture: 
	imageTexture.allocate(imageW,imageH,GL_LUMINANCE);
	
	// set the pixels to be white
	for (int i = 0; i < (imageW * imageH); i++){
		imagePixels[i] = (255); 
		
	}



Exercise 2 - create a static (non-moving) image synthesized using 2-d access of the image’s pixels array  / Download Source Code
 
	texture 2 example

// set the pixels via 2D for loop.... ! 
	for (int i = 0; i < imageH; i++){
		for (int j = 0; j < imageW; j++){
			imagePixels[i * imageW + j] = (i+1)%(j+1);  
		}
	}




Exercise 3 - create a changing (moving or responsive) image synthesized using 2-d access.  / Download Source Code
 
	texture 3 example

for (int i = 0; i < imageW; i++){
	for (int j = 0; j < imageH; j++){
		//calculate the distance from the mouse:
		float diffx = mouseX - i;
		float diffy = mouseY - j;
		float dist = sqrt (diffx*diffx + diffy*diffy);
		if (dist < 60){
			imagePixels[j*imageW+i] = 255+(i+1); 
		} else {
			imagePixels[j*imageW+i] = 0; 
		}
	}
}


    

Exercise 4 - create a drawing tool based on the last example code (drawing into an image whose pixels are being modified) / Download Source Code
 
	texture 4 example

	
for (int i = 0; i < imageW; i++){
	for (int j = 0; j < imageH; j++){
		
		//calculate the distance from the mouse:
        
		float diffx = mouseX - i;
		float diffy = mouseY - j;
		float dist = sqrt (diffx*diffx + diffy*diffy);
		if (dist < 10){
			imagePixels[j*imageW+i] = ((i*255)^(i+100)); 				
		} else {
			imagePixels[j*imageW+i] = MAX(0,imagePixels[j*imageW+i]+1); 
		}	 
	}

    

Week 1 // The 8 Types of Illusion

	
    
    	
  1. Production: Rocco Silano's Miser's Dream Coin Production Magic
    2. 
		
  2. Vanishing: Criss Angel makes his cat disappear
    3. 
		
  3. Transformation: David Blaine turns a homeless man's coffee into money
    4. 
		
  4. Restoration: David Copperfield Destroys Wayne Gretzky's Honus Wagner Card
    5. 
		
  5. Teleportation: Criss Angel Teleports
    6. 
		
  6. Levitation: David Blaine Levitates and then acts real creepy
    7. 
		
  7. Penetration: Liquid Penetration
    8. 
        
  8. Prediction: Stephane Vanel gives away a secret
    9.