Skip to content

Commit 060edde

Browse files
RDAngelFishRDAngelFish
authored
Add files via upload
1 parent 23bf203 commit 060edde

File tree

1 file changed

+146
-0
lines changed
  • arc_design_contest/2019/NCKU_New_Vision_World

1 file changed

+146
-0
lines changed

‎arc_design_contest/2019/NCKU_New_Vision_World/README.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# New Vision World
2+
Welcome to New Vision World!
3+
:smile:
4+
5+
We are ready to show you how New Vision World works.
6+
7+
8+
# Contents
9+
10+
- [Introduction](#Introduction)
11+
- [Function](#Function)
12+
- [Architecture](#Architecture)
13+
- [Hardware and Software Setup](#Hardware-and-Software-Setup)
14+
- [Required Hardware](#Required-Hardware)
15+
- [Required Software](#Required-Software)
16+
- [Hardware Connection](#Hardware-Connection)
17+
- [Component detail](#Component-detail)
18+
- [User Manual](#User-Manual)
19+
- [Before Running This Application](#before-running-this-application)
20+
- [Run This Application](#run-this-application)
21+
22+
# Introduction
23+
24+
Previously, musicians need to have an instrument to perform or practice, and today hand movement of many virtual instruments are detected by wearable devices. With the rapid development of technology, the image recognition of people’s movements and postures is becoming more and more accurate. Therefore, our team hopes to get information of hand movement based on image-related technology without additional wearable devices to let users play the virtual instrument more comfortably. Finally, designing an IoT process to combine more possibilities between technology and multimedia.
25+
26+
## Function
27+
View of piano from user:
28+
29+
<div align="center"><img width="330" height="200" src="images/demo_viewer.JPG"></div>
30+
31+
Three instruments including piano, drum, and guitar viewed from HoloLens:
32+
<div align="center"><img width="300" height="220" src="images/demo_bystander.jpg"></div>
33+
<div align="center"><img width="850" height="200" src="images/demo_instrument.jpg"></div>
34+
35+
36+
37+
## Architecture
38+
39+
System begins in Kinect camera to get RGB information of image. Through the open source called OpenPose, we can generate skeleton of body and keypoints of two hands. Then the computer outputs these information to ARC and doing SVM algorithm to verify each instrument with many different sounds. After getting the result of verification and sounds, we output the data to HoloLens to have a real stage to perform every instuments.
40+
41+
* The main flow is following:
42+
43+
<div align="center"><img width="800" height="300" src="images/dataflow.JPG"></div>
44+
45+
# Hardware and Software Setup
46+
47+
## Required Hardware
48+
* ARC IoT Development Kit (IoTDK)
49+
* ESP8266 Wi-Fi module
50+
* FTDI FT2232 USB to UART module
51+
* Camera (Kinect or any other webcam)
52+
* Client (HoloLens, etc.)
53+
54+
## Required Software
55+
* ARC GNU Toolchain
56+
* OPENNI (for kinect camera here)
57+
* OpenPose (please click [here](https://github.com/CMU-Perceptual-Computing-Lab/openpose) for much more detail requirements), and check GPU performance [here](https://docs.google.com/spreadsheets/d/1-DynFGvoScvfWDA1P4jDInCkbD4lg0IKOYbXgEq0sK0/edit#gid=0). We use GTX 1080 in this project.
58+
* [LIBSVM](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) (a library for Support Vector Machines)
59+
60+
## Hardware Connection
61+
1. Connect Kinect camera to PC (USB port) and power-supply
62+
2. Connect ESP8266 Wi-Fi module to ARC
63+
3. Connect ARC IoT Development Kit (IoTDK) to PC (USB port) and power-supply
64+
4. Connect FTDI FT2232 USB to UART module to PC (USB port)
65+
5. Connect HoloLens to communicate with ARC
66+
67+
* Pin of ARC, p_mod and Wi-Fi module:
68+
<div align="center"><img width="900" height="400" src="images/pin.jpg"></div>
69+
70+
71+
* Real connection between ARC and FTDI and Wi-Fi module:
72+
<div align="center"><img width="400" height="350" src="images/ARC_connection.JPG"></div>
73+
74+
## Component details
75+
76+
* Kinect for Xbox 360 (Kinect) camera
77+
78+
Kinect is created by Microsoft and it has three lens. We use kinect to get real-time RGB information for each image. Before we driver kinect, we use the library called Open Natural Interface(OpenNI) to finish driving.
79+
80+
* ARC
81+
82+
See [toolchain](https://github.com/foss-for-synopsys-dwc-arc-processors) and GNU Toolchain for much more detail.
83+
84+
* USB-FTDI module
85+
86+
It needs to transfer data between the computer and ARC through USB. We choose FT2232HL chip to transfer. [See the document we use.](https://www.intra2net.com/en/developer/libftdi/download.php)
87+
88+
* WIFI – TCP/IP
89+
90+
The WIFI module we used is ESP8266. ARC is the srver and HoloLens is the client. When HoloLens connect to ARC, ARC will transfer 32-bits data contiuously to HoloLens. The flow is following:
91+
92+
<div align="center"><img width="500" height="300" src="images/WIFI.png"></div>
93+
94+
* HoloLens
95+
96+
Microsoft HoloLens is an Augmented Reality (AR) device developed by Microsoft. It is part of the Windows Mixed Reality AR Platform incorporated with Windows 10 OS. Similar to other OHMDs (optical head-mounted displays), HoloLens is a see-through display worn over your eyes. Unlike the Oculus Rift and other VR Devices, the eye-piece component of HoloLens is transparent and the headset requires neither PC nor smartphone. It is able to project high-definition (HD) virtual content or holograms over real world objects.
97+
98+
<div align="center"><img width="200" height="90" src="images/HoloLens.jpg"></div>
99+
100+
101+
* OpenPose
102+
103+
[OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose) represents the first real-time multi-person system to jointly detect human body, hand, facial, and foot keypoints (in total 135 keypoints) on single images. It has been widely used in many applications. Here, in order to verify different instruments, we gonna use body and hands information to do SVM training and testing.
104+
The skeleton of body and keypoints of hands are following:
105+
106+
<div align="center"><img width="330" height="250" src="images/OpenPose_skeleton.png"></div>
107+
108+
* Support vector machines(SVM)
109+
110+
SVM is a discriminative classifier formally defined by a separating hyperplane. In other words, given labeled training data (supervised learning), the algorithm outputs an optimal hyperplane which categorizes new examples. In two dimentional space this hyperplane is a line dividing a plane in two parts where in each class lay in either side. We train model with 200 data per class.
111+
112+
# User Manual
113+
114+
## Before Running This Application
115+
* You need to install (Most software are implemented on python and you need to have GPU 1080 up):
116+
1. ARC GNU Toolchain
117+
2. Third party on python
118+
* OpenPose (please click [here](https://github.com/CMU-Perceptual-Computing-Lab/openpose) for much more detail requirements)
119+
* Pybind for FTDI: Download (path) to finish binding c program and python to do FTDI and more detail of FTDI FT2232 driver with [libftdi](https://www.intra2net.com/en/developer/libftdi/index.php)
120+
* OpenNI for Kinect: You can include (/OpenNI) to your file or go to [github](https://github.com/avin2/SensorKinect) for more detail.
121+
* HoloLens: Refer to the [HoloLens tools](https://docs.microsoft.com/en-us/windows/mixed-reality/install-the-tools) site for relative files.
122+
* [LIBSVM](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) for SVM
123+
* Thread for running program parallel
124+
* Playsound for playing music
125+
126+
## Run This Application
127+
1. Run ARC and initial
128+
129+
![ARC_start](images/ARC_initial.JPG)
130+
131+
2. ARC connects FTDI to communicate with PC
132+
133+
![ARC_start](images/ARC_USB.png)
134+
135+
3. ARC connects Wi-Fi to communicate with HoloLens
136+
137+
![ARC_start](images/ARC_Wifi.JPG)
138+
139+
* Through HoloLens connects to ARC, it will show a debug box and a buttom of connection to ARC
140+
141+
![HoloLens_connection](images/HoloLens_connection.jpg)
142+
143+
4. Run python code to send keypoints from PC to ARC
144+
145+
146+

0 commit comments

Comments
 (0)