Без темы
<<  Li-Fi: световая замена Wi-Fi MICE-возможности эмирата Абу-Даби  >>
LightSync: Unsynchronized Visual Communication over Screen-Camera
LightSync: Unsynchronized Visual Communication over Screen-Camera
2D barcodes everywhere
2D barcodes everywhere
http://www
http://www
A video of barcodes instead
A video of barcodes instead
Visual communications on phones
Visual communications on phones
Many challenges
Many challenges
In the rest of the talk…
In the rest of the talk…
Frame synchronization challenges
Frame synchronization challenges
Frame rate mismatch
Frame rate mismatch
Received frame patterns
Received frame patterns
Received frame patterns
Received frame patterns
Received frame patterns
Received frame patterns
Received frame patterns
Received frame patterns
Display frame rate
Display frame rate
Synchronized communication
Synchronized communication
Unsynchronized communication
Unsynchronized communication
Requirements
Requirements
Our solution: LightSync
Our solution: LightSync
LightSync Demo
LightSync Demo
LightSync
LightSync
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Mixing patterns
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
Per-line tracking: Reference blocks
LightSync
LightSync
Inter-frame coding
Inter-frame coding
Inter-frame coding
Inter-frame coding
Performance
Performance
Implementation
Implementation
Performance
Performance
Schematic performance
Schematic performance
Schematic performance
Schematic performance
Schematic performance
Schematic performance
Schematic performance
Schematic performance
LightSync performance
LightSync performance
LightSync performance
LightSync performance
LightSync performance
LightSync performance
Conclusion and outlook
Conclusion and outlook
Thank you
Thank you
Related work
Related work

Презентация: «LightSync: Unsynchronized Visual Communication over Screen-Camera Links». Автор: Wenjun Hu. Файл: «LightSync: Unsynchronized Visual Communication over Screen-Camera Links.ppt». Размер zip-архива: 4212 КБ.

LightSync: Unsynchronized Visual Communication over Screen-Camera Links

содержание презентации «LightSync: Unsynchronized Visual Communication over Screen-Camera Links.ppt»
СлайдТекст
1 LightSync: Unsynchronized Visual Communication over Screen-Camera

LightSync: Unsynchronized Visual Communication over Screen-Camera

Links

Wenjun Hu Hao Gu Qifan Pu

2 2D barcodes everywhere

2D barcodes everywhere

3 http://www

http://www

windowsphone.com/zh-cn

(“Linking” to online information)

4 A video of barcodes instead

A video of barcodes instead

“Transmitting” information (vs linking) E.g., information on Gibraltarian attractions E.g., movie show times for nearby theaters

Receiver

Transmitter

5 Visual communications on phones

Visual communications on phones

No need for additional connectivity Free, directional, lightweight links “Optical” software-defined radio Run as apps on commodity devices

6 Many challenges

Many challenges

Frame synchronization Poor image quality Limited computation power on phones Aesthetics…

7 In the rest of the talk…

In the rest of the talk…

Frame synchronization challenges LightSync design Implementation and evaluation

8 Frame synchronization challenges

Frame synchronization challenges

9 Frame rate mismatch

Frame rate mismatch

Screen side (TX)

Camera side (RX)

High rate, steady

Low rate, unsteady

30 fps video playback Fairly stable

15 – 30 fps in practice Variable Lighting conditions Device capability API/system factors

10 Received frame patterns

Received frame patterns

Original frame

11 Received frame patterns

Received frame patterns

Original frame

Single frame

2-frame mix

Mixing pattern varies by line

12 Received frame patterns

Received frame patterns

Due to Rolling Shutter

CMOS camera sensor scans one line (of pixels) at a time

13 Received frame patterns

Received frame patterns

Original frame

Single frame

2-frame mix

Consecutive frames indicating a missing frame

3-frame mix

Mixing pattern varies by line

14 Display frame rate

Display frame rate

Occasional missed frames

4-frame mixes

2C

C – Frame capture rate

Occasional missed frames

C

C/2

15 Synchronized communication

Synchronized communication

Under-utilized and inefficient!

Display frame rate

2C

C – Frame capture rate

(of worst camera)

Cannot be used

C

The only decodable pattern Frame synchronized

Carries information But undecodable and dropped

C/2

16 Unsynchronized communication

Unsynchronized communication

Works with any display and capture rates Higher camera capture rate, better performance

Display frame rate (fps)

30

0

15

Supported frame capture rate (fps)

30

17 Requirements

Requirements

Need to be able to decode mixed frames In-frame color tracking per line Need to guard against lost frames/lines Inter-frame erasure coding Essentially a temporal code across frames Works with any (monochrome) per-frame code

18 Our solution: LightSync

Our solution: LightSync

Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding Essentially a temporal code across frames Works with any (monochrome) per-frame code

19 LightSync Demo

LightSync Demo

20 LightSync

LightSync

Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding

21 Per-line tracking: Mixing patterns

Per-line tracking: Mixing patterns

3 frames mixing ? 3 blocks mixing Bl or w per block ? 8 mixing possibilities

?

?

?

?

RX

TX

Frame 1

Frame 2

Frame 3

22 Per-line tracking: Mixing patterns

Per-line tracking: Mixing patterns

TX

RX

Frame 1

Frame 2

Frame 3

Line 1

Line 2

Line 3

23 Per-line tracking: Mixing patterns

Per-line tracking: Mixing patterns

Trivial cases

TX

RX

Frame 1

Frame 2

Frame 3

Line 1

Line 2

Line 3

24 Per-line tracking: Mixing patterns

Per-line tracking: Mixing patterns

Easy cases

TX

RX

Frame 1

Frame 2

Frame 3

Line 1

Line 2

Line 3

25 Per-line tracking: Mixing patterns

Per-line tracking: Mixing patterns

Tricky cases, to be tracked per line

TX

RX

Frame 1

Frame 2

Frame 3

Line 1

Line 2

Line 3

26 Per-line tracking: Reference blocks

Per-line tracking: Reference blocks

Code area

Tracking bars

Corner

Corner

Corner

Corner

27 Per-line tracking: Reference blocks

Per-line tracking: Reference blocks

4 reference blocks per line

Tx frame 1

Tx frame 2

Tx frame 3

Tx frame 4

Tx frame 5 (repeats)

Tracking bar 1

Tracking bar 2

Tracking bar 3

Tracking bar 4

28 Per-line tracking: Reference blocks

Per-line tracking: Reference blocks

Tx frame 1

Tx frame 2

Tx frame 3

Tx frame 4

Tx frame 5 (repeats)

Tracking bar 1

Tracking bar 2

Tracking bar 3

Tracking bar 4

29 Per-line tracking: Reference blocks

Per-line tracking: Reference blocks

Tx frame 1

Tx frame 2

Reference colors of mixes

Tx frame 3

Tx frame 4

Decoding by comparison

Tx frame 5 (repeats)

Tracking bar 1

Tracking bar 2

Tracking bar 3

Tracking bar 4

30 LightSync

LightSync

Need to be able to decode mixed frames In-frame grayscale tracking per line Need to guard against lost frames/lines Inter-frame erasure coding

31 Inter-frame coding

Inter-frame coding

Simple pairwise XOR

Group of 3 original frames

Coded frames

32 Inter-frame coding

Inter-frame coding

Looping display on screen

All original frames

Coded frames

Camera stops frame capture after message decoded

33 Performance

Performance

34 Implementation

Implementation

Inter-frame intervals

Current frame capture rate

35 Performance

Performance

Goal recap: Works with any display/capture rate combinations Metric: Effective frame capture rate Actual throughput scales with per-frame capacity

36 Schematic performance

Schematic performance

For a 20fps phone camera

37 Schematic performance

Schematic performance

For a 20fps phone camera

Ideal

Ideal

Peaks at the camera rate

Level at higher display rates

38 Schematic performance

Schematic performance

For a 20fps phone camera

Ideal

Conventional

Peaks at ~ half the camera rate

Drops to 0 at higher display rates

39 Schematic performance

Schematic performance

For a 20fps phone camera

Ideal

LightSync

Conventional

Peaks at the camera rate

Slight drops at higher display rates

40 LightSync performance

LightSync performance

Nokia Lumia 920 (~30 fps)

41 LightSync performance

LightSync performance

Nokia Lumia 920 (~30 fps)

HTC One X (~20 fps)

42 LightSync performance

LightSync performance

Works with any display and capture rates Higher camera capture rate, better performance

Nokia Lumia 920 (~30 fps)

HTC One X (~20 fps)

Huawei Ascend C8812E (~15 fps)

43 Conclusion and outlook

Conclusion and outlook

LightSync: Unsynchronized visual communication Primitive for similar scenarios Lots of potential in the space of visual communication

44 Thank you

Thank you

wenjun@microsoft.com

45 Related work

Related work

Temporal barcode design (inter-frame) Unsynchronized 4D Barcodes Basically slows down display, only for TX up to 10fps Spatial barcode design (per-frame layout) COBRA: Optimized for phone-to-phone PixNet: High-end camera, computationally intensive, offline processing Other work on visible light communications

«LightSync: Unsynchronized Visual Communication over Screen-Camera Links»
http://900igr.net/prezentacija/informatika/lightsync-unsynchronized-visual-communication-over-screen-camera-links-237191.html
cсылка на страницу

Без темы

778 презентаций
Урок

Информатика

130 тем
Слайды
900igr.net > Презентации по информатике > Без темы > LightSync: Unsynchronized Visual Communication over Screen-Camera Links