Department Of Electrical And Computer Engineering

Data Acquisition Glove for Portable Electronic Sign Language Translation

by

Garry Rank

A thesis submitted for the degree of

Bachelor of Engineering (Computer Systems Engineering), Bachelor of Science (Computer Science)

Synopsis

A data glove design has been researched as part of an overall system with the purpose of translating Australian Sign Language into speech electronically. The M16C/62 development board transmits to a PC detailing finger bend and general contact data, which can then be displayed on screen. Common micro-controller options of analogue to digital conversion and port direction are the core of the system.

Introductory Letter

209 Kent St

Karawarra, Perth, W.A

6152

November 5th 2004

To the Head of Department of Electrical and Computer Engineering, Curtin University of Technology, Professor Syed Islam

This is the thesis covering the topic of

Data Acquisition Glove for Portable Electronic Sign Language Translation

for the course of Bachelor of Engineering (Computer Systems Engineering) / Bachelor of Science (Computer Science) (double degree), and as a partially satisfying requirement for that course.

I certify that this thesis is entirely my own work, except where referenced and / or outside acknowledgement given.

Signed: _____________________________________________

Acknowledgements

I would like to thank Jenny Reid for invaluable assistance in getting me started on this project, without which I would have been lost for some time.

Thanks also to Craig Newman, without whose invaluable time programming the M16C/62 development board would have taken much longer, and for contributions and ideas along the way.

In addition, thanks go to Mike Malak, again for much input and ideas over the course of this project.

And, of course, thanks go to my project supervisor, Iain Murray, for his time, input, motivation and help throughout the year, with everything from the project design to checking on this thesis.

Nomenclature

ADC: Analogue to digital converter – an electrical component to take a digital reading of an analogue voltage.

ASL: Australian Sign Language, also sometimes used for American Sign Language. A national form of sign language.

BSL: British Sign Language – the sign language form as practised in Britain.

UART: Universal asynchronous receiver / transmitter – an electrical component on many circuits to allow for serial communications with another device.

Table of Contents

1 Introduction	1

1.1 The Problem	1

1.2 Solution Overview	1

1.2.1 Goals	2

1.2.2 Objectives	2

2 Background	3

2.1 History of Australian Sign Language	3

2.2 Glove Based Input vs Visual Devices	5

2.3 Virtual Reality Gloves	7

2.4 Previous Work	10

3 Data Requirements	11

3.1 Australian Sign Language Details	11

3.1.1 Hand Physics	11

3.1.2 Breaking Down ASL	12

3.2 Glove Requirements	16

4 Glove Design	19

4.1 Overview	19

4.2 Bend Sensors	19

4.2.1 Solutions	19

4.2.2 Optic Fibre	20

4.2.3 Flex Sensor	22

4.2.4 Capacitive	24

4.3 Contact Sensors	26

4.3.1 Types of Contact	26

4.3.2 Primary Contact Sensors	27

4.3.3 Secondary Contact Sensors	32

5 Implementation	34

5.1 Capacitors	34

5.2 Capacitor Testing Environment	40

5.2.1 Hardware	40

5.2.2 Software Development	42

5.3 Contact Sensors	47

5.4 Contact Sensor Testing Environment	48

5.4.1 Hardware	48

5.4.2 Software Development	49

6 Verification	57

6.1 Capacitor Results	57

6.1.1 Raw Results	57

6.1.2 Moving Plates	58

6.1.3 Multiplexing	60

6.1.4 Hand Mounting	60

6.2 Contact Sensor Results	61

6.2.1 Initial Problems	61

6.2.2 UART Timing	62

6.2.3 Sensor Sweep Timing	62

7 Design Refinements	65

7.1 Bend Sensors	65

7.1.1 Glove Construction	65

7.1.2 Capacitor Construction	66

7.1.3 Electrical Interference	67

7.2 Contact Sensors	67

7.2.1 UART Timing Improvements	67

7.2.2 Capacitor Code Timing	70

7.2.3 Overall Timing	71

8 Conclusion	72

8.1 Future Developments	72

8.2 Actual Translation	73

8.3 Alternative applications	74

9 Bibliography	75

10 Appendix A – Project Plans	77

11 Appendix B – Some Signs	78

12 Appendix C – Source Code	79

Illustration Index

Figure 2.1 Exos Dextrous Hand Master (2004)	8

Figure 2.2 Immersion's CyberGlove (2002).	9

Figure 3.1 Hand straightening from a fist. Image adapted from Abrahms & Hutchings (2003, p159).	11

Figure 3.2 Finger abduction muscles (Moore & Dalley, 1999, p770).	12

Figure 3.3 Thumb movements (Moore & Dalley, 1999, p767).	12

Figure 3.4 “Two” (left) and “kneel” (right) signs (image taken from Newman, 2003, p29)	13

Figure 3.5 “Two”, “spoon”, “three” and “mother” signs (image taken from Newman, 2003, p30)	14

Figure 3.6 One handed alphabet E (left) and S (right) adapted from the Finnish one-handed alphabet, but similar to the Irish one-handed alphabet (Some interesting facts about sign language interpretation, 2004).	14

Figure 3.7 Important hand locations, adapted from Johnston (2000).	15

Figure 4.1 Optic fibre cross section, image modifed from Fibre Optic Technologies (1992).	20

Figure 4.2 Flex sensor (image from Flex Sensors, 2003).	22

Figure 4.3 Adapted from Bones and Joints (2004) Left shows vertical compression. Right shows horizontal sliding.	24

Figure 4.4 Cylindrical sliding plate capacitor design.	26

Figure 4.5 Image taken from Newman (2003, p34).	28

Figure 4.6 Contact sensor locations for one hand.	30

Figure 4.7 Auslan two handed vowels, modified from Auslan – Australian sign language (2003).	31

Figure 5.1Voltage fall-off over time for varied capacitors in series with a 1MΩ resistor.	37

Figure 5.2 Voltage levels for various circuit time constants at specified times.	38

Figure 5.3 Sliding capacitor mounted on soccer goal keeping glove	39

Figure 5.4 Finger bending slightly	40

Figure 5.5 Finger bending more	40

Figure 5.6 ADC wiring to capacitor	41

Figure 5.7 M16C/62 development board attached to a sliding plate capacitor	42

Figure 5.8Capacitor software testing flowchart.	44

Figure 5.9 Multiplexing input test wiring	47

Figure 5.10 Contact sensor testing glove.	48

Figure 5.11 Contact sensor sweep first cycle flowchart.	54

Figure 5.12 Contact sensor sweep second cycle flowchart.	55

Figure 6.1 Conversion results (20 values per result column) of percentage of plate overlap vs ADC determined values.	58

Figure 6.2 Dynamic capacitor test with plates at full overlap.	59

Figure 6.3 Dynamic capacitor test with plates moved slightly	59

Figure 7.1 Coiled capacitor design showing anode on the outside and cathode on the inner track. Dielectric and insulation not shown.	66

Figure 7.2 Non-splayed fingers.	69

Figure 11.1 Auslan two-handed alphabet (Auslan – Australian sign language, 2003).	78

Index of Tables

Table 5.1 ADC Control Register settings	46

Table 5.2 Interrupt Timer Register Settings	56

Introduction

The Problem

For most people verbal communication is the primary form of information exchange and conversation. Be it booking a seat at a restaurant to simply talking to a friend over the phone, much of human interaction is through the ability to speak. For many people this can be a problem; those born deaf and who have never heard speech in order to learn it or those who are mute and can not speak instead rely upon alternative forms of communication – predominantly sign language.

Being reliant upon sign language to speak contains inherent drawbacks, typically when trying to talk to people who do not understand sign language. An interpreter is used in these cases. This places financial constraints upon the signer, and there is not always an interpreter when needed.

The solution to this problem is a cost effective, non-obtrusive device for sign language interpretation on the level where two people can communicate.

Sign language itself is interpreted through visual recognition and much research is being done into a purely CCD based interpretation system, however this is a long term goal. The current level of technology and portable computing power is unable to handle this. As a result, additional data acquisition instruments must be used to supplement that obtained by a visual device.

Solution Overview

Goals

The obvious instrument to use in order to supplement a CCD camera is a sensor glove such as those used in virtual reality applications, and is the focus of this thesis. Any actual pattern recognition is beyond the scope of this document, rather this is a study on the acquisition of required data.

The primary goal of a sensor glove is to determine hand positions – amount of bend of the fingers, and if they are touching anything, the bend of the wrist – not only for one hand, but two handed interaction. This is aimed to be achieved through the usage of cheap equipment without a loss in quality. Hand location with reference to the rest of the body is not required.

Objectives

In researching development of a sensor glove, the following plan was executed in order to further the feasibility and basic design of a sensor glove:

Understand the basic concepts of Australian Sign Language, and what hand and finger motions and positions are required for recognition of signs.

Determine cost and ability effective methods of data capture in a glove design.

Test the validity of researched designs by choosing and using appropriate equipment for testing purposes.

While these points are few, they never the less are broad in their very nature. The final results were not completed in terms of a fully functional glove, however each element was proven to work within acceptable limits for a real time data acquisition glove.

Background

History of Australian Sign Language

Almost in its entirety, this section (2.1) has been paraphrased or quoted from Signs of Australia on CD-ROM (Johnston, 2000).

Auslan, or ASL, has its roots in British Sign Language. It was not created by any person or committee but rather evolved and continues to grow like any spoken language in use today.

With British colonisation of Australia, British Sign Language (BSL) of the early 1800's was used. Large schools were created for the deaf and the form of sign language learned there has had the greatest influence on Australian Sign Language development. Since then Auslan and BSL have drifted apart as languages but still remain closely related.

In addition to British settlers there were a large proportion of Irish immigrants who used their own one-handed alphabet and other Irish background signs (which in turn have roots in French and other European nation sign languages). The private Catholic schools used many of these signs with Protestant schools using predominantly ASL as derived from BSL, including a two-handed alphabet. The two educational societies were not segregated and often mixed. This has led to many modern Auslan signs based heavily upon Irish Sign Language including the modern Irish one-handed alphabet (although the two-handed alphabet is used more regularly).

Auslan today differs across Australia. Much as you can tell where someone lives by they way in which they pronounce place names and slight changes in accent, so can the same be said for different forms of Auslan. The different forms do not seriously impede communication. Two separate schools set up in 1860 are the major contributing reason for two major Auslan dialects, with smaller differences between states. The Sydney school was founded by Thomas Pattison, “a deaf man educated at the Edinburgh Deaf and Dumb Institution” (Johnston 2000) and the Melbourne school was founded by Frederick Rose, a deaf man “who was educated at Old Kent School, London”. These two men had different backgrounds in signing and so passed their own learnings through their respective schools. As the dominant schools for sign language in Australia each had a large impact upon signing as their areas of influence grew, and variations in Auslan can be traced back through the schools' expansion.

There are also differences in signs between ages groups. Auslan is a changing language and schooling systems also change the focus of learning (for example, the amount of importance placed upon finger spelling). Changing technology may also mean younger people use different signs where their generation are more familiar with the technology.

Today there are three main drives for the changes in Auslan, (Johnston, 2000).

Attempts to standardise usage.

The development of new signs to meet new needs.

The borrowing of signs from other sign languages (almost exclusively from American Sign Languge).

The points are the result of the growth of need for interpretation services in various schooling levels, government, legal and medical areas. For many words, especially in scientific circles, there does not exist a sign or adequate alternative to the words. Many signs must therefore be simply made up or used from other languages. While in general signers have no qualms about borrowing signs from American Sign Language where they are required, it is not used where there is already a perfectly suitable sign in Auslan.

In many cases a single Auslan sign can be translated by many different English words and the way in which a sign is performed can also change the meaning. For example if the sign for “work” is used in a hard a sharp manner it may mean the signer had a hard day's work, where as if the sign were performed in a rather loose manner it may mean they had an easy day's work or that it really wasn't work at all. Auslan is, like spoken English, based heavily upon the context in which it is used.

Glove Based Input vs Visual Devices

Australian Sign Language, like any other sign language, is visual based. For a full understanding of a signer there is more to it than simply recognising hand shapes and their position with relevance to the rest of the body. That being said, the hand shapes are also vital to the language. For this reason hand shapes and location have typically fallen into two categories in the past: visual based data acquisition and worn input device data acquisition.

Visual devices typically use a camera to analyse each image and extract necessary data. In the case of hands and fingers this may still require a glove with tracking LED's or colour patches on them. This allows the user to not become encumbered with large devices and by using CCD cameras can be quite portable and durable. Problems do exist in that one camera can only capture 2D information but sign languages depend upon full 3D movement. There is also the case of computing power. Today's computers are simply not powerful enough to provide a real time, portable solution. Some elements of Auslan could not be determined electronically any other way (facial expression for example) but given the current levels of technology alternatives can be used to supplement the visual devices.

Worn input, typically a glove such as those used in virtual reality applications, can be used for fast hand shape interpretation. Many virtual reality gloves also include gyroscopes of some form or another to determine the pitch, roll and yaw of the hand or infra-red devices to capture hand location information. This takes a large bulk of computing power off the visual devices and can be attached to laptop computers easily. Problems that exist with glove based solutions is they they require the user to wear bulkier items and not everybody has the same hand shapes or sizes. A glove that fits one user and detects their hand shapes to a high degree of accuracy may be too large of small to fit another hand. This may not only give inaccurate readings, but damage the glove itself.

As technology increases to a level where the entire sign recognition is feasible through purely visual based devices the need for gloves can be phased out. In the mean time this is simply not an option and any translational device will require a glove of some sort to be used in a real time sense.

Virtual Reality Gloves

Data input gloves were first design for use in virtual reality environments. One of the first glove built for this industry was design for musical use where the glove would detect hand movements of a conductor and feed them through a synthesizer. With the gaming industry booming at the time Matel soon produced the Power Glove for game input. From here it was only a small step to start developing gesture recognition devices for use in games, robotic control and communication between non-vocal people.

Some of the more advanced and accurate virtual reality gloves use an exoskeleton mounted over the hand and arm with various tracking devices and bend sensors mounted upon it. While these undoubtedly provide excellent resolution at high speeds of gesture recognition, they take a relatively long time to put on, may intrude with hand movements (especially involving two handed interaction), and due to weight and cabling constraints are generally not very useful in a mobile sense. The Dextrous Hand Master by Exos, shown in Figure 2.1, is one of the first examples of these types of devices.

Figure 2.1 Exos Dextrous Hand Master (2004)

Some other developments in virtual reality systems allow for a much more compact solution. One example here is the CyberGlove created by Immersion Corporation (CyberGlove, 2002). Shown in Figure 2.2 the CyberGlove is a low profile glove that is produced in 18 and 22 sensor varieties to measure finger and wrist movements. Optional sensors can be attached to track where the hand is. This all comes at a cost of between $20 000 and $30 000 (AUS) which places such systems quite out of reach of everyday users.

Figure 2.2 Immersion's CyberGlove (2002).

For tracking hands in 3D space many companies use devices such as those supplied by Polhemus (http://www.polhemus.com). The Fastrak (Fastrak – the fast and easy digital tracker, 2004.) system is an electromagnetic device that can track in six degrees of freedom through location and orientation, and in real time. A single transmitter can accept data from up to four receivers inclusive, making it useful for two handed data capturing. Such technology is again not cheap, especially if two were required.

Virtual reality gloves also suffer from another drawback that makes most of them unsuitable to sign language translation. They are designed as self contained units. While some may allow the use of two gloves at any given time, any errors in data collection from the fingers and hand may be much greater than a specifically designed two-handed interaction system.

Previous Work

Much of the work done with contact sensors was through an expansion on the work of Craig Newman (Newman, 2003) in his project work entitled “Translating Australian Sign Langage into Speech”. In this project an M16C/62 development board was used to gather information about touching components of the hands and then combine this with a data gather from a P5 virtual reality glove (Newman, 2003, p29) for proof of concept. Some of the issues encountered in Craig's work include:

The P5 glove fell short of being able to gather all necessary data. It could not detect independent bending of the middle and base knuckles.

Hand location tracking was often lost and had to be recalibrated. Background noise interfered with this.

Location with reference to the head and rest of the body could not be obtained if the user moved.

Adaptive algorithms in sign recognition were not used and signs had to be made carefully to be recognised.

Data Requirements

Australian Sign Language Details

Hand Physics

Many sensor gloves are designed with the purpose of obtaining as much information from the hand as possible. This is a waste of data if only those elements important to Australian Sign Language need be considered, however, possible hand movements also help in the understanding of exactly how a sign works with relevance to hand movement and some of the intricacies and subtleties involved in determining hand shapes.

Figure 3.1 Hand straightening from a fist. Image adapted from Abrahms & Hutchings (2003, p159).

Figure 3.1 shows hand movements when making a fist, or straightening a hand from a fist shape. The bend of the base and first knuckles are independent of each other, however the second knuckle of each finger bends with the first, allowing the tips of the fingers to fold into the palm. This also helps to grip objects.

Figure 3.2 Finger abduction muscles (Moore & Dalley, 1999, p770).

Some of the muscles controlling finger abduction (finger splaying), in addition to other movements, are shown in Figure 3.2. Every finger can abduct to some degree or other, with the index and little finger muscular arrangements allowing them more freedom of movement in this respect.

Figure 3.3 Thumb movements (Moore & Dalley, 1999, p767).

The thumb is the most versatile digit on the hand, with some of its movement depicted in Figure 3.3. In terms of sign recognition, it will require the most amount of sensors to accurately depict.

Breaking Down ASL

Actual hand motions used in creating the various signs available are the very core of determining exactly what data is required in electronic sign recognition. ASL takes advantage of all the basic hand movements in signing, summarised below:

Finger Bend – the amount of bend of each finger at the base and first knuckle.

Finger Splay – if adjacent fingers are closed or open.

Finger Crossover – if two, or more, fingers are crossed over one another.

Thumb – palm cross over and finger contact.

Wrist – the amount of bend of the wrist.

ASL is a language of general hand movements, not specifics. How much a finger bends is not of exact interest to sign recognition – rather that it makes the fingers hooked straight is of importance; how the bending affects the location of the fingers and hand in general rather than an absolute value for the degrees by which the finger is bent at that particular knuckle. Examples of a sign changing with finger bend is shown in Figure 3.4.

Figure 3.4 “Two” (left) and “kneel” (right) signs (image taken from Newman, 2003, p29)

Finger splaying is determined in a binary logic format; if the fingers are splayed or not. An example of some signs that depend upon finger splaying are shown in Figure 3.5.

Figure 3.5 “Two”, “spoon”, “three” and “mother” signs (image taken from Newman, 2003, p30)

The thumb provides the greatest flexibility of any digit, and also perhaps the most subtle in terms of sign difference. One example is the difference between the one-handed alphabet characters of 'E' and 'S' as shown by Figure 3.6.

Figure 3.6 One handed alphabet E (left) and S (right) adapted from the Finnish one-handed alphabet, but similar to the Irish one-handed alphabet (Some interesting facts about sign language interpretation, 2004).

Here it can be seen that whether the thumb is over or under the other fingers changes the meaning entirely. It is also worth noting here that this change in thumb position also requires subtle changes in the other fingers.

Much of ASL is through two handed interaction with the right and predominant hand moving to some contact on the left. The contact areas of importance are shown below in Figure 3.7.

Figure 3.7 Important hand locations, adapted from Johnston (2000).

The palm and top of the hand, the top and bottom of the wrist, to the left and right of the hand and around the fingers. Many signs require sliding motions, for example sliding the right hand index finger along the index finger of the left hand. Sign interaction is not based upon static hand locations alone – sliding and various contacting on different locations of the hands can all contribute to a single sign.

In addition to the general areas are the specifics of one finger touching another from the other hand, for example the index finger of the right hand touching the little finger from the left hand (with that hand's thumb tucked in) to represent “fourth”.

Any combination of hand movements as noted above interact differently to produce different signs and meanings. Noted here also, but beyond the scope of this aspect of glove design, is that pointing to various parts of the body, and moving the hands around different areas must also be taken into account when signing. Included here is the direction of which the palm of the hand faces. These aspects of sign language are more easily picked up by a CCD camera than a glove as they rely upon movements of the arm in general rather than being specific to the hand.

Glove Requirements

For any determination of a particular sign, the most obvious requirement is timing. The sensor glove must be part of a real time system that detects and processes information fast enough such that no sign is missed. With so much of any sign language dependent upon very subtle movements, any of these movements not detected would change the meaning substantially.

Assuming a higher end speed of 4 short signs per second (using the one-handed alphabet for example) this gives 0.25 seconds per sign, with as much as half of that time belonging to transient hand positions from changing signs. This gives a very minimum of 8 samples per second, however the effect of error margins should also be taken into account. Placing a limit of 80% accuracy upon the system for correct sign interpretation, this now gives 10 samples every second, or one entire data set acquisition and processing every 0.1 seconds. This is a feasible target given modern day technology even with the use of development boards. It should be noted however that as part of a broader system, speeds greater than this may be required to ensure that overall real time speeds are not affected.

Accuracy is also of crucial importance to the sensor glove. While Auslan itself deals with general hand movements, such as if the fingers are hooked or not, individuals will differ in the amount of bending of each finger for any particular sign. For other signs, again using ‘E’ and ‘S’ as examples, the amount of bending at a particular knuckle can change by only a small amount, but may be the key factor to removing any ambiguity from sign detection. If each knuckle can rotate the next finger segment by approximately 90°, then detection of every 22.5° may be sufficient for general translation, but due to the smaller movements of some signs an accuracy of every 10° is more desirable.

While speed and accuracy are the primary design concerns, the glove would also have to be robust. It is expected that any glove used for these purposes should not have to treated with the utmost of care and should be able to withstand the impacts and forces of constant hand movement. When a signer is angry, they may tend to strike their hands together more forcefully and any device breaking as a result would not only cost the signer to replace it, but most probably aggravate their temper as well. While it also should be expected that the system not have to contend with somebody purposefully destroying it, the design will have to minimise the effects of general wear and tear.

The goal of the glove design is a sub-system to a sign language interpretation system. There should not be the need for much change to adapt to it on the part of the user. This can not be expected to be removed fully while retaining high accuracy, but the amount of adapting should be kept to a minimum. One effect of this is that the glove should have a low profile and minimise interference with any hand movements the user may make. In addition, if the glove does not interfere with hand movements, hand movement will not interfere with proper working of the glove, increasing the robustness of it.

The system overall is aimed as a general translation tool for anybody to use, and to accommodate this, the system should be affordable. Inexpensive technology is required to form the basis for the glove design, minimising the cost of both the glove any eventual system overall. Prices of computing technology is also rapidly decreasing and many items that may prove to be costly now may not be so in the future.

Glove Design

Overview

The glove design can be broken down into two main areas:

Bend sensor development.

Contact sensor.

Part of the system as a whole is knowledge of the position of the hand with relevance to the rest of the body; if the hand is at head height, where it is around the torso, if the hands are crossed over and so forth. The use of on-glove gyroscopes was researched, however the responsiveness of such a system is either too low or too high and LEDs picked up by a CCD provide a far more effective method. Less prone to error, less obtrusive and a cheaper cost make this the favoured approach and so is left to the workings of a CCD.

Bend Sensors

Solutions

The idea of virtual reality gloves is not a new one, and appropriate technology to detect the amount of finger bending has been researched in the past. Methods of finger bend detection are varied, with some more suitable to the role than others. Most virtual reality gloves utilise either of optical fibre, flex sensors (a commercial sensor designed for flex detection) or other patented technologies unavailable for public viewing.

Optic Fibre

Optical fibre bundles have heavy use in the communications industry for their high bandwidth data capabilities. In ideal conditions, signals sent down a straight optic fibre line have little loss over large distances. As soon as the fibre is bent, however, then signal strength degrades – a property which makes another use of it for bend sensing. To understand this better it must first be understood how optic fibre technology works.

Figure 4.1 Optic fibre cross section, image modifed from Fibre Optic Technologies (1992).

An optical fibre strand consists of a hollow glass or plastic wire at the core with a protective cladding wrapped around it. The cladding also serves another purpose, and its refractive index is chosen to be less than that of the core. A beam of light is sent down the wire at an angle such that instead of refracting through the glass and cladding, the difference in the refractive index between the two mediums causes the light beam to reflect instead. Figure 4.1 depicts a typical cross-section of optical fibre. This causes the signal (light beam) to bounce down the wire. Being a light signal, this method is also somewhat fast, but suffers from various drawbacks if not constructed properly.

If the light beam is not entered at an appropriate angle, instead of reflecting down the wire the angle of incidence causes it to refract instead – the light passes through the sides of the optical fibre without bouncing down it towards the other end.

Another property of optical fibre is that it can be bent without breaking. This provides a key usefulness in installing it for communications as cables must often be bent around intervening objects. Different grades of optic fibre are available, and some are more prone to breaking than others at various amounts of angular stress. This ability for it to bend is also another factor that can cause light signals to not hit a surface at an angle of incidence low enough to be reflected instead of refracted.

In theoretical environments a single light beam is either reflected or refracted and mathematical equations can be used to determine the exact angle necessary. In practice a single light beam can not be used. A focused source of light is sent down the fibre, but some energy will be lost to absorption by the mediums used, and slight leakage of light from the mediums due to impurities (moisture in the air, water left over from manufacturing). One result of this is that there is no definitive cut off between light being reflected and refracted – instead signal power at the receiving end is gradually decreased with increasing bending of the fibre. This of course tends favourably towards its use as a bend sensor; the more the fibre is bent, the less the signal power being transmitted. A photosensitive diode or other such electrical component can be used in conjunction with an analogue to digital converter to determine the amount of transmitted power.

As a bend sensor optical fibre lines have a very high degree of possible accuracy, and the speed at which they operate is limited only by the timing of a photosensitive device and analogue to digital conversion timing. It should be noted here that the degree of accuracy possible is not required – far less will be sufficient for the system. For a robust system optical fibre lines may not be as appropriate, depending upon materials used, as they can be prone to breaking with constant use. The cost of repairs to such a device are conceivably high, and in order to increase the lifetime of the device care would have to taken in its use. These design limitations make other methods of bend sensing more suitable than optic fibre. In addition there are patents upon virtual reality gloves using such technologies which incur increased costs and legality issues if it were to be used.

Flex Sensor

Widely used today in applications from virtual reality gloves to detecting if a (usually robotic) device has collided with an object (such as a wall), flex sensors are a commercially available product which change resistance as the sensor itself is bent. An image of one can be seen in Figure 4.2 with circuit connection pins on the left.

Figure 4.2 Flex sensor (image from Flex Sensors, 2003).

These devices utilise a flexible plastic for the base and a string of pockets with carbon ink between metal plates. The carbon ink acts as are resistive element between the metal contacts. When the sensor is bent the resultant forces pull the metal contacts apart and spreads out the conductive ink more. With the ink spread out more the actual short circuit path of voltage through the ink is increased, which is an increase in resistance of the sensor. This resistance is of approximate linear fashion with the degree of bending of the sensor.

Through manufacturing experience, flex sensors have become quite reliable and allow for repeated use without breaking. The speed of fluid movement with movement of the sensor is high enough for use in a virtual reality glove, although actual timing requirements are unknown. As a user of sign language could be moving their hands rather rapidly, such information is highly important. Another design limitation with this type of sensor is that it detects one direction of bending only and bending the fingers back may produce confusing results for an interpretation system. This problem may be sidestepped by simply attaching the sensor so that it is straight when the fingers are bent back, however different people have different hand sizes and degrees to which their fingers may move back, and the flex sensor is designed for angles between 0° and 90°.

Another down side to flex sensors in a sign language interpretation system is their standard length of production, typically around 4.5”. This is long enough to cover the base knuckle, but may also cover the next knuckle and beyond. Bending at one knuckle can then interfere with the data read from another. From some manufacturers this can actually be used to an advantage as contacts along the length of the flex sensor strip can be connected to in order to allow the resistance at various points to be determined. In doing so, the number of required flex sensors for the glove may be reduced.

Their width and thickness also make flex sensors very unobtrusive, and by using velcro can be fine tuned easily to a person's hand. Not only would this allow for more accurate readings from any individual, but can lessen the effects of glove material pulling at the sensor and possibly damaging it, or pulling apart connections soldered to it. Other causes of errors in readings can be from crossing the fingers over and two handed interaction – the flex sensors are susceptible to pressure upon them which can be achieved either through twisting (as may occur from crossing the fingers) or direct force (pressing down on that area from two handed interaction).

Being a commercial device, flex sensors cost money. While they do have the reliability, accuracy and potential speed requirements to be used for a sign language interpretation glove, this comes at a cost of (depending upon exchange rates) near $20.00 AUS each. If one is required for each bend location to be detected, 24 are needed for two hands. If one can be used for two knuckles, then this number decreases to 14, but still places the component prices range of the flex sensors alone between $280 and $480. In bulk, flex sensors may be cheaper.

Flex sensors can be used as an accurate, robust solution to the problem of bend sensing, but other methods may prove more appropriate – only creating and testing glove prototypes can determine this.

Capacitive

Other methods described here work upon the basis of the sensor itself bending with the finger. There are other properties of the hand that make alternative approaches viable as well, namely what actually occurs when a finger bends.

Figure 4.3 Adapted from Bones and Joints (2004) Left shows vertical compression. Right shows horizontal sliding.

As can be seen from the above figure 4.3 two different properties of finger bending can be used to base a sensor upon. In Figure 4.3, left, it is shown how vertical compression can be achieved.

Two strips are attached to a glove in the fashion shown such that they are not allowed to slide. As the finger is bent, the lower layer bends with the finger and stretches a small amount due to its distance from the centre of curvature. The outer layer is further from the centre of curvature than the lower layer and so the endpoints will move a further distance; though their angular displacement be the same, the radius from the centre of the knuckle causes a greater requirement of distance to move by the same angle as the lower one. The only way for the material to accomplish this (assuming it to be non-elastic) is move down (with reference to the lower strip), hence vertical compression occurs. Moving the fingers to a straight position releases the tension on each strip and they may vertically separate.

Vertical compression is the basis for the flex sensor workings. The compressive forces spread out the conductive ink as explained above. Another electrical component that could be modified to suit these purposes is a capacitor. By changing the distance between capacitor plates the electrical properties of a circuit can be predictably modified. This idea was soon abandoned as it was discovered that keeping the capacitor in place over a knuckle is difficult and would have to modified on a per person basis – a difficult prospect at best.

Another aspect of finger bending is shown by Figure 4.3, right. Here there are again two strips: one attached to the top of the hand, the other allowed to slide and attached (by some material) to the finger above the joint. When the finger bends the top strip is allowed to slide. The reasons for this are similar to vertical compression where the top strip is attached at some distance from the centre of rotation. For the same angular displacement a greater distance must be covered, and if rigidly attached then the only way this can be achieved is if the strip slides forward or back depending upon which direction the finger is moving. This lends itself naturally to a capacitor solution where the strips present are capacitor plates. As one plate slides over another, then the surface area where one plate overlaps the other is changed, which in turn affects the capacitor value.

A capacitor plate need not be flat. Cylindrical shapes can be used to incorporate the housing around which a plate is required to slide. Figure 4.4 [note which plate slides] shows the fundamental design concept of this. Smaller components are desirable as the glove itself should be unobtrusive and not impede hand movements.

Figure 4.4 Cylindrical sliding plate capacitor design.

Contact Sensors

Types of Contact

In two handed interaction the types of contact to be expected from each hand can be broken down into primary and secondary considerations. Primary contacts are those relating to contact of on part of one hand touching another part of the same or other hand. Examples of this include fingers touching the palm, fingers crossing over and finger splaying. All of these actions are localised to the hands and deal directly with the shape of the hands and digits and in addition to providing necessary information which bend sensors can not always provide, can increase speed of sign recognition through redundant data (for example, the amount of finger bend is unimportant if it is known a finger is touching the palm of the hand). Secondary contacts are used more for increased speed of sign recognition for the system as a whole, where contact with body parts such as the arms or head should still be detected as involving the hands, but are not as crucial to knowing what a sign is. An important note here is that with increased accuracy for CCD facial, body and hand location data acquisition, the need for any secondary sensors diminishes and so are not discussed in great detail.

Primary Contact Sensors

Contact sensors that deal directly with hand shapes should not be affected by movement from other parts of the body, or contact with other parts of the body. To this end, elements such as a simple button to detect if the finger is pressed against something become unsuitable. The sensors should only be triggered when pressed against an appropriate hand area.

Much of the thought behind this has already been researched by Craig Newman (Newman, 2003, p33) and involves the use of external metal contacts placed around appropriate locations of the hand as opposite sides of a switch. The contacts are separated into output and input points – writing high to an output allows all contacts classed as inputs to be read as voltage high when they are touching an output. Only one output is written high at a time so it can be determined to which contact they are touching. Resistors connected to ground provided protection for the microchip as well as ensuring that inputs were read as voltage low (0V) when not connected. A diagram of this basic configuration is shown in Figure 4.5.

Figure 4.5 Image taken from Newman (2003, p34).

This design offers high sensitivity for contact detection. A simple brush against an opposite contact will be detected, a desirable trait as it then does not require the user to use any additional force to which they are accustomed when pressing against the appropriate areas. It also allows detection of only glove based events; pressing the fingers against other areas of the body will not affect the results. There are drawbacks to this approach, but the benefits of simplicity and effectiveness of design outweigh these.

Some drawbacks to the contact approach include damage to the contacts and interference from external influences. As the contacts must be located on the outside of the glove, they must also be situated above all other sensors, including the bend sensors. Movement of the hand could warp metal contacts, so if used they would have to be kept clean to prevent dirt from producing false readings. Touching objects and general wear and tear may dent or otherwise damage the contacts, especially if they are metal, and there is the concern of touching parts of the body with live contacts. The only way to overcome the latter concern is to use minimal voltage and current levels to ensure the human body is not used as a conductive path. At levels used by most microchip circuits, the power levels are not enough to harm a person, if they are able to be detected by the person at all.

Warping of the contacts can be avoided if metal is not used for the contacts. One conductive material capable of bending with the hand as it moves is carbon rubber. Flexible enough to be used and though not as conductive as metal contacts (such as brass), still provides sufficient conductivity to be useful as an external contact material (carbon rubber is in fact used for push buttons such as those found on calculators). Various grades of carbon rubber are available to increase flexibility or conductivity as desired.

The locations to place sensors is also important. Some areas of the hand do not require contacts to be located upon them where other areas are of critical importance and do require them. It is desired to minimise the contacts required as this not only decreases complexity of the overall system but also decreases the chances of any part of it becoming damaged.

Obvious locations are the tips of the fingers and the palms of each hand. From Figure 3.5 other important locations are the tops of the palms, top and bottom of the wrist and either side of the hand. Contacts can also be used between fingers for fast determination if the fingers are spread apart or not, and determining if two hands are interlocked or not. While interlocked fingers are not often used in Australian Sign Language, they are used and so must be covered by the glove. This configuration is summarised in Figure 4.6. 24 contacts per hand, 48 in total, can cover all information necessary. Some of these contacts may not be required by some signers, but would be required for others. Only prototype testing on various people would determine which sensors, if any, are actually not required (for example the contact on top of the little finger may not be necessary).

Figure 4.6 Contact sensor locations for one hand.

Contact sensors developed by Craig Newman could have been used for two handed interaction but were primarily tested for one hand. Two handed interaction is possible with the existing configuration but can not cover all circumstances, such as fingers of opposite hands touching. An example of this is shown in Figure 4.7 with the two handed vowel alphabet signs.

Figure 4.7 Auslan two handed vowels, modified from Auslan – Australian sign language (2003).

To properly meet glove design requirements slight modifications are needed. The first response is to simply add more sensors to the glove. Drawbacks of this approach are increased complexity of the system and greater chance of one sensor interfering with correct operation of another. An alternative solution to this problem is by using the same number of sensors but simply switching the direction of output to input, and input to output for the pins connected to the contacts (thus changing polarity of the contacts). This is possible on most microchip families which contain I/O pins by simply changing a few register values. Of course this takes a small amount of time to propagate throughout the system, but should only be required twice every data collection cycle (once to switch each port, and once to switch back). Depending on the software developed a worst case scenario of switching it once for each glove could exist but this would only introduce a three phase switching cycle (switch first glove, switch both, switch back second glove).

Another effect that must be taken into consideration with this type of system is signal bounce. Affecting almost all switch based input into digital circuits the idea of signal bounce is where a closed switch may bounce by a very small amount in the physical sense. The switch may alternate between an open and closed state for a small period of time which while typically unnoticeable for a light switch or other simple circuit on / off application, can be detected by digital circuits. The typical solution to bouncing effects is to either modify the RC value of the circuit such that a slow discharge rate cancels out bounce effects, or or use a time delay in software. Software is the preferred method today as it requires less elements to be incorporated into the hardware.

Other effects generated by contact sensor circuits can include capacitive and inductance effects. When power is cut off to the sensors it may take a small amount of time to dissipate power charge depending upon the resistance in the wires and the size and distance apart of the contacts. There is also that case that when sending a current through a closed circuit that the circuit may be in some form of loop. In this case there is a change in current for a small period of time, which can create a magnetic field and so induce current into nearby wires. This can produce false readings indicating other contacts are closed when in fact they may not be. The typical solution here is the same as for debouncing the circuit; introduce a time delay before taking any readings.

Secondary Contact Sensors

As stated previously, secondary contact sensors are used for better determination of a sign for when the hands are touching areas other than a glove. If a hand is known to be in the general vicinity of the chest and the secondary contacts are touched, then it can be implied that the hand is touching the chest. Such information may seem redundant at first, as sign languages are based upon general area information and will not require the user to actually touch their chest, but depth calculations from CCD cameras can suffer from a high degree of inaccuracy which could be alleviated from secondary glove sensors. These accuracy problems will be lessened with more research and development of image manipulation from visual based translation devices to the point where secondary sensors are simply not required.

Several options are available to use for secondary contacts. As they do not require only glove triggers, they may be enclosed and so protected from damage better than primary contacts. They will also not be as sensitive to light contact and so while providing possibly redundant information for fast sign determination, it should not be used as an absolute; just because the sensor is not triggered does not mean a person's hand is not touching an area – the touch may simply be too light to be detected by the sensor.

Simple buttons may be used as secondary sensors, such as those found in calculator key panels, or commercially available pressure pads that change resistance when pressed. Pressure pads are widely used in virtual reality gloves for their increased sensitivity over a simple button, and can also be used to cover a wider area. Simple buttons also work best when force is applied directly perpendicular to the contacts, which is not always the case for a sign language. Pressure sensors cover this facet more adequately and so become the better solution for secondary sensors.

One commercially available pressure sensor ideal for placing onto gloves is the Flexiforce Pressure Sensor which changes resistance depending upon the force applied to it. The cost of these are relatively high.

Implementation

Capacitors

One of the design requirement is that the glove be unobtrusive. To accommodate this, capacitors should not be too large. Copper was chosen for the conductive plates due the ease with which it is soldered to, with small sheets containing enough tensile strength to remain undistorted by movement when attached to the glove. A plastic sheath from a file binder was used for a dielectric to place between the copper plates both for its availability and dielectric value (of approximately 2). It was also thin which allowed for the copper plates to be placed closer together, increasing the capacitance.

The copper was cut to 25mm by 15mm sized plates. This size allows the plates to slide with hand movements, without areas of no overlap. If a range existed for no overlap of the plates, this would give the same readings and defeat the purpose of a bend sensor.

Looking at the formula for determining capacitance:

Where C is the capacitance

є is the permittivity of free space (approximately 8.85x10-12)

A is the effective surface area of plate overlap

d is the distance between capacitor plates

k is the dielectric constant of the material.

Given a width of 0.2mm for the plastic dielectric, this gives the constructed capacitor 26.55pF at full plate coverage to 13.275pF at half plate coverage. Measured values agreed with these results within experimental error and were 27pF at full plate coverage, and 13pF at half plate coverage. Any differences are explained by slight variations between the actual size of the copper plates and any capacitive effects introduced by wire soldered onto each plate.

It was noticed that contact with the human body greatly increased the value of capacitance. This is because of slight conductivity of the skin and an increased effective surface area of opposite plates. The increase is substantial – as much as one thousand times higher readings were seen. Bend sensors based upon sliding capacitor plates are required to be properly shielded from any contact with the hand, however this should be done anyway to prevent damage to the sensors and so was not perceived to be a problem.

One method of measuring capacitance levels of a circuit is by using AC signals and reading back any signal shifts. A resistor would be required in series with the capacitor so that the shift is increased with increasing capacitance, and decreased with decreasing capacitance. Most microchips do not support features such as these, instead relying upon DC circuit properties for pin inputs. Properties of capacitors in simple RC circuits (with DC voltages) is that they are treated as open circuits after an initial charge period. If the voltage source is turned off, the capacitor will discharge. The rate of charge and discharge is known as the time constant of the circuit – or its RC value (being Resistance x Capacitance). The formula for determining the voltage level of an RC circuit at any given point in time is:

Where t is the time (in seconds).

A 1MΩ resistor was chosen to be placed in series with the capacitor. This resulted in a minimum 13μs time constant, which should be enough such that the analogue to digital converter of a microchip can sample and note the value. As the design is also part of a real time system, time spent waiting for charge and discharge of a capacitor should be kept to a minimum.

Graphs of voltage levels after source turn off are shown in Figure 5.1. In order to properly determine changes in capacitance, the points of highest voltage difference were desired in order to give the most accurate readings. It was also realised that time would be taken to perform any analogue to digital conversion, and the range over which largest differences occur were predicted to balance out the readings; if the voltage level drops at a rate consistent with the amount of capacitor plate overlap, then while it would affect readings, they would still be affected by an amount relative to how much the finger is bend, and so produce a usable scale. A relative scale for changes in finger bending was the goal, not the actual value of capacitance at any given time.

Figure 5.1Voltage fall-off over time for varied capacitors in series with a 1MΩ resistor.

There is also the problem that these differences in voltage are not of linear scale. Figure 5.2 depicts voltage levels for changing RC values of the circuit at various points after the voltage source is turned off. For RC values between 10μs and 30μs it can be seen that an approximate linear relationship can be used for t = 30μs. It is at this level that the voltage was desired to be read, as it produces the most linear scale of amount of bend.

Figure 5.2 Voltage levels for various circuit time constants at specified times.

Different capacitors were created in order that they could be better attached to the hand. They were constructed of brass tubing 4cm in length, with size 5/16 for the outer cylinder and 1/4 for the inner cylinder. Plastic as used before formed a separating barrier for the plates. When tested for actual capacitance values it was realised that the distance between the inner and outer plates more than accounted for any increase in surface area, with the capacitance level reaching a maximum of 13pF at full plate overlap. This made the them unsuitable for use in testing, though were still of use in determining difficulties in attaching capacitors to a glove.

A pair of soccer goal keeping gloves were bought to mount sensors upon. As these were soon modified to the point of being unsuitable for contact sensors to be attached, football grip gloves were also bought. Sports gloves were the item of choice as they are specifically designed to fit the hand without restricting its movement, and without sliding about. This would minimise any problems with glove material movement giving erroneous readings.

Figure 5.3 Sliding capacitor mounted on soccer goal keeping glove

Mounting one capacitor on the soccer gloves can be seen in Figure 5.3. Velcro was used to attach and reposition one of the cylindrical capacitors to the glove on the top of the hand, with an old computer mouse cord used to slide the inner cylinder. The mouse cord was attached by velcro to a finger so that it too could be repositioned to fit over a knuckle properly. Figures 5.3, 5.4 and 5.5 show the capacitor sliding as the finger is bent.

Figure 5.4 Finger bending slightly

Figure 5.5 Finger bending more

Capacitor Testing Environment

Hardware

The M16C/62 was chosen as the test bed due to its high clock rate (16MHz), onboard analogue to digital conversion units, high number of programmable I/O pins, UART capabilities, comprehensive manuals, and ease of programming. With the development tools for the microchip based around the C programming language, this was the obvious choice in which to code software.

Figure 5.6 ADC wiring to capacitor

Wiring the test together is shown in Figure 5.6. Note that due to the open circuit effect of capacitors on a DC circuit, the resistor is placed before the capacitor in the circuit. The voltage probe shown is an analogue-to-digital converter input pin on the M16C/62 development board, and the +5V source one of the output pins – writing an output pin logical high results in +5V at the pin, logical low is not guaranteed to 0V but is generally close enough to be assumed so.

A standard PC system was chosen for the data collection point. This was simple to connect to the development board via the included serial cable and provided an excellent means for viewing real time data. The cygwin environment for windows (http://www.cygwin.com) was used for the PC software development for both the low cost (it is freely available) and UNIX structure that meant any code should require minimal changes to work on any computer. Standard POSIX serial communication libraries were used to retrieve data from the development board.

The primary testing device is shown connected to the M16C/62 development board in Figure 5.7.

Figure 5.7 M16C/62 development board attached to a sliding plate capacitor

Software Development

The primary requirement of software for the M16C/62 development board was not efficiency of memory or general speed, but rather the verification that capacitive differences could be quickly detected. For ease of program creation and testing, the basic configuration used by Craig Newman (Newman, 2003, pp138-191) was stripped down and modified to work with the analogue to digital converter (ADC).

A basic software requirement is listed below:

Initialise an output pin to high. Enough time must be allowed to pass before setting this pin low in order to allow the capacitor to charge to full potential.

Set output pin low. The capacitor will start to discharge.

Wait 30μs (or close to) for voltage levels to drop.

Start analogue to digital converter. Greater lengths of time taken here may produce skewed results, so timing should be kept to a minimum.

Wait for conversion to complete. The M16C/62 contains register flags that indicate ADC status.

Read conversion results. These are typically stored in separate registers for each ADC pin. 16 bits are available for each register to allow 10bit conversion mode values to be stored.

Enable output pin to high, allowing the capacitor to recharge. It is not important that the capacitor is fully discharged, only that it is allowed to fully charge.

Transmit results to PC. Before this can be done, the results must be placed into an effective transmission format, typically by sending one byte at a time.

This simple configuration is illustrated in Figure 5.8.

Figure 5.8Capacitor software testing flowchart.

As speed and accuracy were the testing parameters of the design it was important to understand what settings should be used. According to the specifications given by Renesas Technology Corporation (Renesas, 2004) the smallest possible conversion time is using 8bit mode at the CPU's main clock speed. This gives a minimum conversion time of 1.75μs when operating at 5V device input and 16MHz clock speed. The actual conversion is run in sample and hold mode where the voltage is sampled over 3 cycles of the ADC clock (CPU clock speed in this case) and conversion run on the sample. If sample and hold functionality is not enabled, the conversion timing is increased.

Speed is not the only requirement of the micro-controller; accuracy is just as important. When running in 8bit resolution mode only 255 levels above 0V to 5V are available. At 30μs after voltage to the capacitor is turned off, the input to the ADC pin is anywhere between 0.25V and 1.75V, a range of only 30% when compared to a full 0V to 5V range. This in turn gives only 76 different possible values obtained from the ADC value registers. Assuming an accuracy of every 10º of bend is required and only 90º variations are handled, this gives a range of 8 values per 10º. Given the nature that 3 samples are required to perform digital conversion of the inputs, and various other electrical inductance noise in the signal, this does not leave a large room for error. Larger widths per 10º help alleviate this problem which leaves 10bit operation mode decided for use in analogue to digital conversion.

In 10bit resolution mode, 1023 levels above 0V are available. Using similar calculations to 8bit mode as discussed above, 10bit mode gives 306 different values for a 90º range (or 34 values per 10º). This does nothing to prevent noise in the line, but it does give more accurate readings of the voltage and a greater error margin (a small amount of noise will not produce as large a difference in digitally calculated values) compared to 8bit resolution. The increase in conversion timing is not substantial and takes a minimum of 3.3μs, only 0.5μs more than 8bit resolution mode. The increased accuracy of 10bit resolution far outweighs any timing drawbacks and was deemed to be the best solution when using the M16C/62 – other microchips offer various other ADC resolutions, but none were available for testing.

Pins belonging to port10 of the M16C/62 are used for the analogue to digital converter subsystem. Register values and what they should be set to are shown in table 5.1.

ADC Control Register Settings
ACCON0	0x00	Pin AN0 selected.
ADCON1	0x30	10bit mode, Frequency fAD selected.
ADCON2	0x01	Sample and Hold conversion method.

Table 5.1 ADC Control Register settings

Pin AD0 (pin 0 of port 10) was used for the input to the analogue to digital converter, and pin 1 of port was used for the voltage source.

The M16C/62 has only 8 pins available for analogue to digital conversion. This represents a problem in that 24 bend sensors are required, and so some method was required to multiplex three sensors into one pin. When multiplexing inputs, the requirement is that only one is read at a time. There are ample I/O pins available to charge capacitors, however if they are connected together then the results are determined by three different capacitors (and hence three different areas of bending) instead of one. To overcome this inadequacy, the circuit in Figure 5.9 was devised.

Figure 5.9 Multiplexing input test wiring

Although only two capacitors were used in the test, it would still prove if the diodes would prevent capacitors wired to B and C from charging if only A was written high at 5V. Spare diodes were used, however their exact specifications were unknown. Voltage discharge from a capacitor results in an AC signal being produced, and the capacitive effects of diodes must then be taken into account. How much this would affect results was unknown, and so the simplest method to determine this was to wire up the circuit and see.

The PC software did nothing more than output the result of analogue to digital conversion to screen. Each set of two bytes received from the development board was actually a 16 bit number representing the conversion value (of which only 10 bits are actually used). 8 bits high are first read, followed by 8 bits low, from which the 8 bits high are shifted left 8 places, added to the 8 bits low and finally stored in an unsigned short value (being the 16 bit unsigned integer value on the PC used).

Contact Sensors

Shin brass was chosen for the external conductive material of the contact sensors. It was more readily available in shapes that were easier to work with than carbon rubber, is easily soldered to, and malleable enough to shape and fit onto a glove.

The shin brass was cut into rectangular shapes and holes punched around the other edges. This would allow easy sewing to the glove. Hookup wire was used to solder the brass for attachment to various resistors, and pins on the M16C/62 development board.

Only three contacts per hand were made, as this was deemed to be all that was required to determine the suitability of the design. More contacts could have been created, but this would only have served to increase the amount of time spent sewing contacts onto a glove; this was not a final production glove, but the testing of an idea. The testing gloves for this can be seen in Figure 5.10.

Figure 5.10 Contact sensor testing glove.

Contact Sensor Testing Environment

Hardware

The M16C/62 development board was the equipment of choice due to availability and a high number of I/O pins that were capable of being switched from input to output and vice versa on the fly. This would require reprogramming the flash memory from the software used for capacitor testing, however before this could be done the board started behaving peculiarly. When connected to ground the analogue to digital conversion unit would output what seemed to be noise ranging from an effective digital scale of between 0V and 3.75V. The board could not be trusted to work properly for the contact sensors and so no testing was done. Reasons for the unusual behaviour were determined to be either electrical interference, power grid fluctuations or a damaged circuit. Time was unavailable to further test the discrepancies.

Software Development

The software to run for detecting contact sensors seemed straight forward in that a simple cycle of reading port values is the basic design. Complications began to arise when it was recognised that a single contact could be touching multiple other contacts. The prime example is that of a palm sensor where multiple fingers will be touching it. For the case of the palm, this is really redundant information in that if the fingers are touching the palm, then a reading of the palm touching the fingers is exactly the same data. Information that may be required, however, is that of one palm touching another.

It was theorised that no more than two contacts would touch another at any given point in time. An example is that of a finger touching the palm and being touched by another finger. If for each contact point there are two spaces available in memory to store what that point is touching then a complete picture should be able to be built. This will incur much data redundancy which could then be used to sort through possibly conflicting results to gain an accurate depiction of the current state of each hand. There is also the consideration that increased data per sensor takes longer to transmit. Port I/O switching is the primary goal of the contact sensor testing here, and so each contact will only have stored one other contact to which it is touching.

Debounce and steady state effects must be taken into account when detecting if contacts are closed or not. Fortunately these can be covered by the same code by running a small loop that does nothing more than waste time. This is done before any sensor readings and is generically referred to as a debounce delay even though it may be used for situations other than circuit switch debouncing.

Exact timing of debounce and circuit steady state delays are dependent upon the compiler and how much it will optimise an empty loop. The following pseudo code can be assumed to run at the assembly level:

Loop

Load variable i

Branch to Exit if i == (number of times to loop)

increment i

Loop

Exit Code

return from module.

Taking the core loop code as the main delay, four cycles are required for each cycle of the loop. Depending upon the exact optimisations, ordering of the code within each cycle and instructions available to the CPU, only three cycles may be required for each iteration of the loop. If this is the case it would simply be a matter of tweaking the number of times to loop to fit the system.

The CPU of the M16C/62 runs at 16MHz, 5V input. With four cycles per loop iteration, a time of 0.25μs per iteration is determined. Debounce times for this circuit were estimated to take a magnitude of not more than 0.1ms which results in 400 as the number of times to loop. A value of 1000 was therefore chosen to be tested and increased or decreased depending upon results as it would allow for slightly higher interference and steady state timing. The same debounce time was used through all places in the software, but it may prove more efficient to use separate delays for separate actions (allowing pin I/O change propagation delay to differ from voltage high and low circuit steady state conditions for example).

Ports 2 and 3 were chosen for using with contact I/O as they are not connected to any other peripheral devices or on-chip sub-systems. Pins 0 and 1 of port 2 are set to output and connected to the palm sensors. Pins 0, 1, 2 and 3 of port 3 are set to input and used for the finger contacts. There is no reason that port 2 could not handle everything alone, but in a more advanced system multiple ports and would be required and the testing was designed to simulate those conditions as much as possible. The M16C/62 contains inbuilt pull-up resistors which are not required in this section. It was wanted that the input pins read low when not connected, for which external pull-down resistors were attached, and so the pull-up resistors are disabled.

UART initialisation follows the same path as for the capacitor testing environment. Transmission only is used (though receiving is enabled) and the baud rate is set to 9600. 8bit data transfers with one stop bit and no parity were set. An interrupt timer was set to transmit data at constant times.

After initialisation, the following basic software loop was followed:

Disable UART transmission.

Scan contacts and update data.

Enable UART transmission.

If UART transmission should have occurred during scan, transmit data.

Note that an extra point to transmit data is included. As the system makes continuous scans of the sensors the transmit interrupt may occur during this period and is quite likely to do so. The main task of the microprocessor is to gather data in real time which means 90% of the time it will be doing precisely that. The solution is to simply transmit the data immediately following a sensor sweep. Disabling and enabling of UART transmission is achieved implicitly if the only place the data transmission function is called is by the main loop checking after each full sensor scan.

With six sensors (one for each palm and two fingers from each hand) an unsigned char byte array of length six was used. The following were designated for hand location points:

Sensor 1 - left hand palm.

Sensor 2 - right hand palm.

Sensor 3 - left hand index finger.

Sensor 4 - left had middle finger.

Sensor 5 - right hand index finger.

Sensor 6 - right hand middle finger.

C array indexing means that sensor 1 occupies sensor_map[0], sensor 2 occupies sensor_map[1] and so on. At each array location the number to which that sensor is touching is stored. For example if sensor_map[3] equalled 1 this would indicate the left middle finger was touching the left hand palm. This is by no means a perfect, or even fully adequate, design to use in a proper system but served to show how to go about creating software more more suitable gloves and some of the pitfalls that can be avoided.

The sensor scan is separated into two sections: cycle one and cycle two. Before cycle two is when the I/O directions are switched for selected sensors. For the first cycle the following outline was used:

Clear all previous readings. Allows each sensor sweep to start from scratch and not be influenced by previous results.

Enable left palm contact voltage high.

Debounce delay.

Determine input pins with voltage high as a result.

Set left palm contact voltage low.

Set right palm contact voltage high.

Debounce delay.

Determine input pins with voltage high as a result.

Set right palm contact voltage low.

Change right glove palm to input, right glove fingers to output.

With the flowchart shown in Figure 5.11.

Figure 5.11 Contact sensor sweep first cycle flowchart.

Before the second cycle is initiated another debounce delay is called to allow for I/O switching to complete. Cycle two is similar to the first one.

Enable right hand index finger voltage high.

Debounce delay.

Determine input pins with voltage high as a result.

Set right hand index finger voltage low.

Set right hand middle finger voltage high.

Debounce delay.

Determine input pins with voltage high as a result.

Set right hand middle finger voltage low.

Set left hand palm voltage high.

Debounce delay.

Determine input pins with voltage high as a result.

Set left hand palm voltage low.

Reset I/O pins back to their original state.

The flowchart is shown in Figure 5.12.

Figure 5.12 Contact sensor sweep second cycle flowchart.

After the second cycle another debounce delay is called to allow for I/O switching to propagate throughout the chip.

When sending data through the UART controller it must be sent one byte at a time. There also exists the possibility of errors in the transmission. There are two methods of dealing with transmission error – request a retransmission of the data or simply ignore it. For now errors are simply ignored as no actual transmission errors were seen between the microchip and PC for the capacitor testing.

A timed interrupt was set to enable data transmission. A 2μs source count was set for the nice number it gave, and a 50,000 cycle count assigned to give an overall interrupt period of 0.1ms (thus allowing 10 data captures per second). It was the only interrupt set and so only required a very low priority (set to 1). A summary of this is shown in table 5.2.

Interrupt Timer Settings
ta0mr	0x80	2μs source count.
ta0	0xC350	Hexadecimal for 50000
ta0ic	0x01	Lowest priority timer set.

Table 5.2 Interrupt Timer Register Settings

Verification

Capacitor Results

Raw Results

Conversion results from the test bed are show in Figure 6.1.On the x-axis is the amount of percentage overlap of the capacitor plates, and the direct conversion results on the y-axis. 20 samples are shown for each amount of overlap, additional samples simply remained within plus or minus one of the stabilised value. The PC software was run separate each time to obtain the samples, and explains why eight readings were required before steady results were obtained. Various serial port initialisation protocols affect the readings, although through slight current inductance, additional power requirements or some other form of interference is unknown. Once the UART initialisation from the PC was complete, the only serial communications are from the board transmitting – this only occurs when the ADC is not operational and so is unknown as to whether it would continue to affect the readings or not. Under the assumption that it did, another noteworthy occurrence from the results is that they are all affected by the same degree of interference and an overall scale to use for detecting the amount of bend can still be obtained.

Figure 6.1 Conversion results (20 values per result column) of percentage of plate overlap vs ADC determined values. The results also show a non-linear scale. This is due to the required timing of 30μs difficult to accurately obtain. Greater knowledge of per-instruction timing is required to design a proper loop that will force the software to wait 30μs, or more trial runs to simply guess at the number of cycles in an empty for loop.

Moving Plates

The original goal of sliding capacitor plates was to determine if they could be used to accurately and quickly determine the amount of bend of a finger, or the wrist. With the differences shown becoming stable after a program initialisation period, it is also possible to obtain angle readings of greater accuracy than 10°, though in doing so the chances of any reading to contain errors increases. As a greater accuracy is not required, anything that could increase error probabilities should be avoided. Despite any problems with non-linearity and sample starting issues, the results have shown that it is quite possible.

Several problems still existed at this stage that needed to be examined, notably that these results were performed under static conditions; the capacitor plates were not moving during readings, and they were not attached to the hand. To test moving capacitors a small program for the PC was made that simply contained a triangle that would rotate clockwise or anticlockwise as the one of the plates was moved. Two screen shots can be seen in Figures 6.2 and 6.3. The amount of rotation was taken directly from the results obtained off the board and so were not constrained to 10° angles. The triangle moved across a range of 0° to nearly 180° with repeatable results as the plates themselves were moved, and only a minor amount of jitter was observed (typically the act of touching the capacitor plates influenced the capacitance level). If a range of greater than 90° was possible with such a simple unshielded design then accuracy to within 10° should not prove to be a problem in a complete glove.

Figure 6.2 Dynamic capacitor test with plates at full overlap. Figure 6.3 Dynamic capacitor test with plates moved slightly

Multiplexing

When it came to testing multiplexed inputs into the one ADC pin, this proved to be unsuitable with the used configuration. Changing the supposedly active capacitor had little affect on the results, where as changing the uncharged capacitor produced more usable, but still error prone, results. This was completely opposite to what was intended, with the answer lying in that the diodes used were effectively acting as 30pF capacitors. This value was greater than those designed, and skewed results. If diodes with far less effective capacitance are used, then the basic design could still work, however a more effective measure, if also more expensive, would be to use CMOS circuits purpose built for the task of analogue to digital conversion. These add to the complexity of the design and increase power requirements if 24 are required, but it does avoid multiplexing problems. Using CMOS circuits also allows paralleled conversion which can also help with timing problems. While each chip will generally be run with a lower clock speed than a main microcontroller, that each can run independently allows overall timing to be decreased. Several assumptions are made here including that the slower clock timer does not affect results and that the changing voltage does interfere with other elements of the circuit through inductance. Sufficient shielding around devices will be required in this case.

Hand Mounting

As noted previously, Figures 5.3 through 5.4 show how capacitors can be mounted to slide with the hand. The inner cylinder moved quite freely, assuming the material used to attach it to the finger was of sufficient strength. In designing and creating this several issues were found. Foremost was that crossing the fingers produced greater sliding motions that would give false readings in a proper system. This was due to the mouse cord itself slightly impeding the ability of the fingers to cross. A design that would require less strength of something over the knuckle is to use a spring type mechanism (be it an actual spring, compressed air, or some other elastic design) to bring the sliding plate back to its starting location when a finger straightens.

An additional problem with the cylindrical capacitors was that their overlap also depended upon material stretch. With the wrist bending this would actually force movement of glove material and hence sliding of capacitor plates independent of the amount of bend of the finger. A more rigid design in key areas of the glove would help alleviate this problem. This would also lessen the impact on readings from the affects of two handed interaction upon the sensors.

Contact Sensor Results

Initial Problems

Results could not be directly obtained from the contact sensors due to equipment malfunctions. Theoretical timing could still be analysed, although it is recognised that these results are based upon untested and non-optimised code.

The major considerations with timing in the contact sensor software development is debounce delays, pin I/O swapping and UART communications timing.

UART Timing

Before it can be known at what times to send data for maximum efficiency it should be known how long it will take to transmit the sensor data. It would also show if ignoring transmission errors affects the speed of the system greatly. The baud rate was set to 9600 for the capacitor plate testing which translates to approximately 1.04x10-4s per bit. No parity bit was used, 8bits for data transfer was set, and a start and stop bit were used. Summing to 10bits in total this would take approximately 1.04ms to transmit. To include rounding errors and time taken for data buffer swapping, 1.05ms is assumed here.

A total of 12 bytes of data was sent each time sensor data was transmitted through the serial connection taking a total time of 12.6ms. For a system that may contain 24 bend sensors and a worst case of 48 contact locations there could be as much as 144bytes of data to be sent each time. At 1.05ms per byte of data this is 151.2ms total for a full sensor scan data transmission. Only six full scans per second could be transmitted at this rate – well under the system requirements of ten full scans per second. Obviously something will have to be changed in the design to overcome this problem.

Sensor Sweep Timing

Within the program that was to be used for testing there are eight calls to the debounce time delay function. If each delay takes 0.25ms then a total time of 2ms is spent simply doing nothing. As a start point to see if further developments would turn out to be feasible timing wise, the highest limit on program execution should be determined.

48 contact sensors are assumed to be placed around two gloves. At the start of each main program loop there are 16 of these that are set to input leaving 32 set as output. If each of these 32 must be cycled through with a debounce a total 8ms delay is incurred. One glove then has all contact I/O directions switched adding another 0.25ms delay. There are now 24 contacts in input mode, and 24 in output mode, and again the outputs are cycled through with debounce and steady state delays. Another 6ms delay total is incurred. I/O directions are then changed on both gloves, although the total number of contacts in output and input modes will remain the same, increasing the overall delay by another 0.25ms. A further delay of 6ms is incurred by cycling through the output contacts. In the final step the glove contacts are returned to their original I/O configuration and the total delay is increased again by 0.25ms. The total summation of these delays is 20ms. All this assumes the debounce time is sufficient, which in all possibility it may not be.

Additional program timing must be taken into account when calculating sensor sweep times. If a high level of program execution time were used and the program still fell within design limits then the system could be considered usable. For a 16MHz chip, 10ms was deemed to be a sufficiently high amount of time to run each main loop cycle. At 62.5ns minimum timing per instruction, 10ms would allow 160,000 instructions at most to be executed in addition to those allowed for debouncing. At first 160,000 may seem to be a large amount, but it is not actual lines of code but rather actual instruction executions per 10ms. On an assembly level call one-to-one mapping it would actually be somewhat lower depending upon the exact internals of the microchip, if data is only written on rising edges, and so forth. To look at it from a different perspective, if on average 5 clock cycles were required for one line of assembly code to complete, and there were 5 lines of assembly per line of C code, that comes down to 6400 lines of C. These numbers are purely an example and have to real world meaning, but do show how little it really is. Even so, a basic unoptimised program to run an unoptimised sensor sweep should not be much larger than 1000 lines of C code (it was a little over 170 for the testing scenario, and that includes heavy commenting) and so program execution time should be able to be achieved well within 10ms. More efficient sensor scanning methods would allow this time to increase.

Design Refinements

Bend Sensors

Capacitor based sensors work as an alternative to some of the more costly commercial bend sensor available, but do have their own set of problems to be overcome. Some of the problems faced with results are plate sliding due to material movement in other bend sensor locations, large enough capacitance available, electrical interference and multiplexing.

Glove Construction

Bend sensors as researched require special needs from the glove to work effectively. Material movement plays a large part in errors from readings and this would need to be minimised. By using a thin layer of plastic over the hands this problem could be decreased. The hardened surface would also help to provide a base on which to mount additional components. It would be preferable to contain all capacitors on the same location (the top of the hand in this case) and simply feed connections used for sliding over the fingers. This prevents the capacitor itself from impeding any hand movements but may require slightly larger housing to be built on top of the hand. Proper construction would also prevent skewing of results if they were touched with any force by the other hand, a required trait in a two-handed based system.

Larger housing makes for a bulkier glove, and so a capacitor based design would have to be manufactured as small as possible. Aside from the obvious weight and movement obstruction problems people will not want to wear a glove that is aesthetically unappealing.

Capacitor Construction

One of the problems faced was of capacitance being too low with the cylindrical construction. This was due to the plates being too far apart. Manufacturing would have to ensure that any moulded design was of the correct size. To help increase the flexibility of the design Figure 7.1 shows another possible alternative to cylindrical capacitors. By using the same amount of volume a spiral shape can be used to increase the surface area of the capacitor, and the amount by which capacitance changes as one of the plates is moved. This has a practicality drawback of increased resistance to sliding that may make it unsuitable and the system easily broken if forced. Proper insulation between each surface of the capacitor would also be required, although this is a manufacturing problem and should be easily overcome (most capacitors on circuit boards use this technique to keep their size small).

Figure 7.1 Coiled capacitor design showing anode on the outside and cathode on the inner track. Dielectric and insulation not shown.

Electrical Interference

AC signals generate many different problems, relating especially to induced currents. As was seen with problems from the ADC component of the microchip the entire glove system is prone to problems from internal and external electrical interference. A properly manufactured system of any device such as a sensor glove would require at least some shielding against these effects.

Electrical interference is one aspect the flex sensors have above a capacitor based solution. By using changing resistance they only require a DC voltage to be used, which has far less problems with timing than capacitors. There is no charge and discharge time, and other components are not affected as greatly. There are still a number of unknowns with flex sensors however – their internal steady state conditions, affects of external pressure on readings, durability and long term accuracy.

Contact Sensors

UART Timing Improvements

As was seen with the timing results of contact sensors, the UART transmission time was the largest concern and needs to be reduced. Various methods are available in order to do this including increased baud rate, only one byte of data per sensor and removal of possibly redundant sensor data.

It was realised that a proper system most likely would not use serial communications as is integrated onto the M16C/62 development board. Purpose built hardware for data buffer storage and wireless communications may remove the need of suspending data acquisition from the sensors and allow parallel operation with transmission to the core translational area. For this reason the possibility of removal of start and stop bits through synchronous serial communications is ignored (they may be required, especially with wireless data transfer systems), but never the less, decreased timing in sensor data would be of benefit to the overall system and should be looked at.

Increasing the baud rate is a possibility to decrease the time spent in transmitting data. The next level up from 9600 baud is 14400. At this level it would only take 100ms for a full sensor scan to be transmitted. Assuming this could be constantly transmitted the limit of real time system is reached, however it would not able to be constantly transmitted due to the buffer data copy delays and sensor sweep timing. The highest baud rate available to the M16C/62 is 31250, allowing for 46.08ms spent is data transmission. This should be sufficient to allow sensor scans and other miscellaneous items to be handled and still allow the required ten data acquisitions per second. There exists a problem with simply increasing baud rate. Errors are more likely to occur at higher data rates and while this has been taken into account somewhat with the (at least) ten cycles per second limit it is not known how much greater the chances are of an error.

With errors in transmissions, and especially noting the above serial communications timing, it is also worth noting that any errors on the PC side should simply be discarded. If the microchip were forced to wait for an acknowledgement before continuing with normal operations there would be much greater difficulties in maintaining ten sensor data acquisitions per second. It is simply not worth any increase in data reliability.

One byte of data for each sensor would halve the load on the serial communications timing. It would require the system to receive each data segment in a pre-determined sequence which may be interrupted if a transmission error were to occur. This would leave the the receiving end expecting data for one sensor when in actuality obtaining it for another. For this reason at least two bytes should be used; one byte to contain the sensor number and the next for the sensor data. To differentiate between the two one bit can be reserved to indicate what type of byte it is. 72 (48 contact and 24 bend) sensors only require 7 bits to represent which then allows 1 bit (in an 8 bit data transfer system) to be used on information such as the byte type. If fewer sensors actually required information to be sent then less bits may be required for representation.

Figure 7.2 Non-splayed fingers.

For 48 sensors there is much redundancy information included if every sensor contains another sensor it was connected to. Examples of this are with finger splaying where if, as shown in Figure 7.2, the fingers are touching each other and sensor A is touching sensor B, then sensor A will contain that information in the internal program sensor map. Later readings would also show sensor B touching sensor A, however this information is already stored and is simply a waste if it is sent again. Following this through it can be seen that as much as half the data from the contact sensors could be saved. There is also the case that data should only be transmitted if it has changed from previous results. If an estimate of half the contact sensors and all bend sensor data changed between any given sensor sweep (although there is no research to back up this estimation, it seems conservative) the total number of bytes can be reduced to 72. For a baud rate of 9600 this would only take 75ms to transmit. This would of course require additional on-chip program time and memory space, however, the benefits far outweigh increased program execution time.

Depending upon bend sensor accuracy, it may also prove that some of the contact sensors are simply not required (such as fingers crossed sensors on the little finger) which would also cut back on the amount of data required to be sent. This last point should not be relied upon however, as Auslan is a language in a state of change and some sensors not required now may be required for future signs.

Capacitor Code Timing

From the implementation and results, capacitor readings can be taken relatively fast. With 30μs timed wait for discharge, 3.3μs wait for readings to be determined, plus any additional timing required for program execution, a higher end boundary of 50μs can be assumed per capacitor. Note here that cycle periods of capacitor charge do not need to be considered – while readings are being taken on one capacitor the rest can be charging. Sufficient shielding can prevent current induction from noticeably affecting the results. A round trip time of 1.2ms should be all that is required to probe all 24 bend sensors.

Overall Timing

Assuming 75ms for data transmission over the serial connection, 20ms for sensor reading delay periods, 1.2ms for bend sensor readings and 10ms for additional overall program execution, this is actually 6.2ms over the limit. The result is that serial communications must be higher than a baud rate of 9600, or that a different form of data transfer be used, preferably as an external memory buffer latched onto the main chip used in the gloves and that does not intrude upon CPU running beyond that. This would allow parallel operations in transmitting data and gathering the next set from the sensors.

Conclusion

The research performed has shown that a data acquisition glove does not need to be created from expensive technologies. A robust, unobtrusive, cost effective and accurate system can be made for not only one hand, but a two-handed interaction as well. Most of the problems faced in using capacitors as alternative bend sensors are manufacturing in nature and from that viewpoint may require additional costs due to the specialised nature of what is required. Other alternatives may be much simpler to attach to a glove and the only real way to see which is better suited in the end is to construct one of each and test them.

The proof of concept has been extended from the work of Craig Newman (Newman, 2003) and how contact sensors can be expanded upon to be used in complex two-handed situations. While actual test results were ultimately not obtained there was no foreseeable reason as to why the hardware would not work in an actual glove. Software development was shown to be the critical area of work required in this area.

Future Developments

The gloves themselves still have many problems that require fixing and there is the need for testing of full sensor gloves as opposed to proof of concept as designed in this research. It may well be that what is fine for a test is completely unsuitable for one reason or another when it comes to an actual production level device. Proper materials still need to be examined for contact sensor (notably various grades of carbon rubber) to test their viability and a method of capturing data from all bend sensors needs to be implemented.

Better optimisation of algorithms in sensor sweeps is also essential future work. Worst case scenarios were looked at to show timing considerations, and with small optimisations how a very large reduction in the amount of data to be transmitted could be affected.

As noted previously, future technologies (especially with mobile computing power) may allow sign language translation without the need for sensor gloves to be used. Such technology does not yet exist however, and if any serious move towards an electronic translation system for the general public was introduced, data acquisition gloves would be required.

Actual Translation

The software for actual sign recognition was not even attempted here. Most projects in the past have used neural networks for this purpose which allows for an adaptive recognition system to be built. It would allow the system to learn over time and become fine tuned to an individual user the longer they use it.

A sensory glove is not the only component to a proper sign language translation system. It requires integration into other components of the system such as hand location tracking and facial feature recognition. Current developments in this area require LEDs to be placed upon the glove, or some other technique that allows a CCD camera to easily discern where the hands are relative to the rest of the body.

Alternative applications

Sensor gloves do not only have applications in sign language translation. Typically industrial applications would not require a portable system, relying instead upon high degrees of accuracy from which optical fibre solutions have proven to be effective in the past, but as a cheap device for everyday PC computers there is a range of applications for which it would prove useful. Computer game inputs and motion capture for computer graphics applications could all benefit from a system designed to be two handed, accurate and robust. Even the simplest of task such as file browsing may be found easier by many people if they simply had to point and move their fingers.

Bibliography

16-Bit, 100 kSPS PulSAR® Unipolar CMOS ADC, 2003, Retrieved: October 18, 2004, from http://www.analog.com/UploadedFiles/Data_Sheets/36777272089356AD7660_d.pdf

Abrahams, P., Marks, S. & Hutchings, R., 2003, 'Hand' in Human Anatomy, 5th Edition, Elsevier Science Limited, Edinburgh, p159.

ADC1005 10-Bit µP Compatible A/D Converter, 2003, Retrieved: October 18, 2004, from http://www.national.com/ds.cgi/AD/ADC1005.pdf

Angular displacement sensors, (n.d.), Retrieved: August 14, 2004, from http://www.delphion.com

Auslan – Australian sign language, 2003, Retrieved: November 03, 2004, from http://www.wadeaf.org.au/auslan.shtml

Bones and Joints, (n.d.), Retrieved: November 3, 2004, from http://www.physicaltherapywebsites.com/library/ch9/bonesandjoints.asp

Chapman, R., Pedrycz, W. & Rae S., (n.d.), A motorless artificial limb and its control architecture Retrieved: October 18, 2004, from http://www.compusmart.ab.ca/rc/Papers/eccespaper.pdf

Conner, J., 1997, Baud rate, data bits, parity , and all that, Retrieved: October 31 from http://francis.courtois.free.fr/jc1/serial/Basics/BitFormat.html

CyberGlove, 2002, Retrieved: October 29, 2004, from http://www.immersion.com/3d/docs/cyberglove_datasheet.pdf

Dangers of electricity, 1997, Retrieved: October 19, 2004, from http://acept.la.asu.edu/courses/phs110/ds/appendixC.html

Emerson, T., 1995, sci.virtual-worlds glove faq, Retrieved: October 18, 2004, from http://www.faqs.org/faqs/virtual-worlds/glove-faq/

Exos dextrous hand master, (n.d.), Retrieved: November 04, 2004, from http://www.cs.nps.navy.mil/people/faculty/capps/4473/projects/smithml/handMaster.htm

Fastrak – the fast and easy digital tracker, (n.d.), Retrieved: October 29, 2004, from http://www.polhemus.com/fastrak.htm

Fibre Optic Technologies, 1992, Retrieved: October 18, 2004, from http://www.gare.co.uk/technology_watch/fibre.htm

Flex sensors, 2003, Retrieved: October 18 from

http://www.harris-educational.com/Probeware/experiments/TSA/flex-sensors.htm

Flexiforce pressure sensors, (n.d.), Retrieved: October 19, 2004, from http://www.imagesco.com/catalog/sensors/pressure-info.html

Glove based input interfaces, (n.d.), Retrieved: September 01, 2004, from http://www.dei.unipd.it/~cuzzolin/Glove.htm

(appears to be down at time of writing but a similar page without the pictures was retrieved November 03, 2004, from http://www.geocities.com/mellott124/glove1.htm).

HumanGlove, (n.d.), Retrieved: November 03, 2004, from

http://www.tech-bank.com/Technologies/humanglove.htm

Johnston, T., 2000, Signs of Australia on cd-rom, Deafness Resources Australia, N.S.W, CD-ROM

Kadous, W. 1995, Recognition of Australian Sign Language Using Instrumented Gloves, thesis, University of New South Wales, Retrieved: August 13, 2004, from ftp://ftp.cse.unsw.edu.au/pub/users/waleed/thesis.ps.gz

Kadous, W., 2002, Machine language and sign language recognition, Retrieved: October 18, 2004, from http://www.cse.unsw.edu.au/~waleed/gsl-rec/

Moore, K. & Dalley, A., 1999, 'Upper Limb' in Clinically oriented anatomy, 4th Edition, Lippincott Williams & Wilkins, Maryland, pp.763-770.

Newman, C., 2003, Translating Australian sign language into speech, thesis, Curtin University of Technology

Nilsson, J. & Riedel, S., 2001, Electric Circuits, 6th Edition, Prentice Hall International, New Jersey

Peach, J., 2000, High-Five, thesis, University of Florida, Retrieved: October 18, 2004, from http://www.mil.ufl.edu/imdl/papers/IMDL_Report_Spring_00/Peach_Jack/hi5.pdf

Renesas, (n.d.), Retrieved: April 24, 2004, from http://www.M16Canz.com/

Ross, K., 1997, The basics – very basic circuits, Retrieved: October 25, 2004, from http://www.seattlerobotics.org/encoder/mar97/basics.html

Sedra, A. & Smith, K, 1998, Microelectronic circuits, 4th Edition, Oxford University Press, New York

Some interesting facts about sign language interpretation, (n.d.), Retrieved: November 04, 2004, from http://www.lea-test.fi/en/deaf/part1/some.html

Toney, A., (n.d.), A Novel Method for Joint Motion Sensing on a Wearable Computer, thesis, University of Washington, Retrieved: June 03, 2004, from http://gro.hhhh.org/~joeboy/publications/toney_iswc1998_poster.pdf

Appendix A – Project Plans

Original Project Plan

Weeks	Tasks
1 – 5	Research what is needed, and theoretical solutions.
1 – 3	Bend and Pressure Sensor research
2 – 5	Wrist Location and Rotation
5 – 10	Development and testing of theoretical ideas.
5 – 8	Bend and Pressure Sensor Testing
8 – 10	Wrist Location and Rotation
10 – 20	Integration and testing.
10 – 15	Microcontroller Integration and Programming
15 – 20	Additional requirements as per final methods chosen.
The Rest	Thesis write-up.

Modified Project Plan

Weeks	Tasks
1 – 5	Research what is needed, and theoretical solutions.
5 – 10	Development and testing of theoretical ideas – Bend sensor testing.
10 – 15	Microcontroller Integration and Programming
20 – 22	Thesis production – background material research.
22 – 25	Thesis notes and draft writing.
25 – 30	Contact sensor testing (failed)
34 – 39	Thesis full writeup.
40 – Final Day	Thesis proof read and general finish.

Appendix B – Some Signs

Figure 11.1 Auslan two-handed alphabet (Auslan – Australian sign language, 2003).

Appendix C – Source Code

“sensor.c”

Analogue to digital conversion testing program for the M16C/62 development board.

/****************************************************/

/* Program written by Craig Newman (modified by Garry Rank)	*/

/* */

/*	This program will only run in	*/

/*	stand-alone mode, ie will not run in	*/

/*	KD30 debug mode	since KD30 & monitor program	*/

/*	also uses UART1.	*/

/****************************************************/

#include <stdio.h>

#include "sfr62.h"

void main(void);

void initPort(void);

void initProgramTimers(void);

void initUART1(void);

void transmitConnections(void);	// receive interrupt routine -> Tx data

#pragma INTERRUPT transmitConnections

void main(void)

{

int count;

asm("fset I");	// enable interrupts

initPort();	// Turn off pull-up resistors for ports 0 and 2

initProgramTimers();

initUART1();	// Setup serial communications

// If f(Xin) is greater than 10MHz (here it's 16MHz), then it must be divided.

adcon2 = 0x01;

for(count = 0; count < 1500; count++);

adcon0 = 0x00;

adcon1 = 0x28; // Enable Vref and set 10bit operational mode.

ta0s = 1;	// start program timer : timerA0

u1c1 = 0x04; // Enable receive;

while(1) { }

}

void initPort(void)

{

pu00 = 0;	// no pull-up for P0_0 to P0_3

pu01 = 0;	// no pull-up for P0_4 to P0_7

pu04 = 0;	// no pull-up for P2_0 to P2_3

pu05 = 0;	// no pull-up for P2_4 to P2_7

pd0 = 0x02; // Set pins 0 and 1 of port0 to output.

p0 = 0x02; // Set pin 1 to output high, all other pins to low.

}

void transmitConnections(void)

{

// char Rx;

char adResult_low, adResult_high;

int count;

// asm("fset I");	// enable interrupts

/*

while (u1c1 & 0x08 == 0x08)

{

Rx = u1rbl;	// read Rx data

}

*/

// Rx = u1rbl; // Read receive data.

p0 = 0x00; // Turn off voltage.

for(count = 0; count < 5; count++);

adcon0 |= 0x40; // Start A-D conversion.

while(!ir_adic); // Wait for A-D conversion to complete.

while(adcon0 & 0x40); // Redundant waiting. Used here because this definitely works.

adResult_low = ad0l;

adResult_high = ad0h;

p0 = 0x02; // Turn voltage on again to allow capacitor recharge.

u1c1 = 0x01;	// enable Tx

for (count = 0; count < 1500; count++);	// wait for Tx enable to propagate

// while (!(u1c1 & 0x02));	// Wait while data is present in transmit buffer

// for (count = 0; count < 7500; count++); // wait for PC recieve buffer to empty

u1tbl = adResult_high;	// write data to UART1 transmit buffer

while(!(u1c1 & 0x02)) { }

u1tbl = adResult_low; // Low order bits of A-D registor 0 (pin AD0).

while(!(u1c1 & 0x02)) { }

u1c1 = 0x04;	// enable Rx

}

void initProgramTimers(void)

{

ta0mr = 0x80; // XX0X XX00

// |||| |||+- must always be 0 in timer mode

// |||| ||+-- must always be 0 in timer mode

// |||| |+--- 0: pulse is not output at pin TA0out

// |||| | 1: pulse is output at pin TA0out

// |||| | TA0out is automatically output

// |||| +---- 0: gate function: timer counts only

// |||| when TA0in is held "L"

// |||| 1: gate function: timer counts only

// |||| when TA0in is held "H"

// |||+------ 0: gate function not available

// ||| 1: gate function available

// ||+------- must always be 0 in timer mode

// |+-------- count source select bits:

// +--------- count source select bits:

// 00: f1

// 01: f8

// 10: f32

// 11: fc32

ta1mr = 0x80;

ta0 = 0x208D;	// Set program execution to 60Hz

ta1 = 0x0C35;	// Set debounce timer to 6.25ms

ta0ic = 0x01; // ---- XXXX

// ||||

// |||+-- Interupt priority level select bit

// ||+--- Interupt priority level select bit

// |+---- Interupt priority level select bit

// | 000: Level 0 (interrupt disabled)

// | 001: Level 1

// | 010: Level 2

// | 011: Level 3

// | 100: Level 4

// | 101: Level 5

// | 110: Level 6

// | 111: Level 7

// +----- Interupt request bit

// 0: Interrupt not requested

// 1: Interrupt requested

ta1ic = 0x02;	// higher level than timerA0 because it occurs within timerA0 interupt function

}

void initUART1(void)

{

// Setting UART1 transmit/receive mode register (UART mode)

u1mr = 0x05; // XXXX XXXX

// |||| |||+- uart mode

// |||| ||+-- uart mode

// |||| |+--- uart mode

// |||| | 100: 7 bit data

// |||| | 101: 8 bit data

// |||| | 110: 9 bit data

// |||| +---- Internal/external clock select bit

// |||| 0: Internal clock

// |||| 1: External clock

// |||+------ Stop bit length select bit

// ||| 0: One stop bit

// ||| 1: Two stop bit

// ||+------- Odd/even parity select bit

// || Valid when bit 6 = 1

// || 0: Odd parity

// || 1: Even parity

// |+-------- Parity enable bit

// | 0: Parity disabled

// | 1: Parity enabled

// +--------- Sleep select bit

// 0: Sleep mode deselected

// 1: Sleep mode selected

// Setting UART1 transmit/receive control register 0 (UART mode)

u1c0 = 0x10; // 00XX XXXX

// |||| |||+- BRG count source select bit

// |||| ||+-- BRG count source select bit

// |||| || 00: f1 is selected

// |||| || 01: f8 is selected

// |||| || 10: f32 is selected

// |||| || 11: inhibited

// |||| |+--- /CTS//RTS function select bit

// |||| | (Valid when bit 4 ='0')

// |||| | 0: /CTS function is selected

// |||| | 1: /RTS function is selected

// |||| +---- Transmit register empty flag

// |||| 0: Data present in transmit register

// |||| (during transmission)

// |||| 1: No Data present in transmit register

// |||| (transmission completed)

// |||+------ /CTS//RTS disable bit

// ||| 0: /CTS//RTS function enabled

// ||| 1: /CTS//RTS function disabled

// ||+------- Data output select bit

// || 0: TxD0 pin is CMOS output

// || 1: TxD0 pin is N-channel open-drain

// || output

// |+-------- Must be fixed to '0'

// +--------- Must be fixed to '0'

// Setting UART1 transmit/receive control register 2 (UART mode)

ucon &= 0x80; // -X0- --XX

//	|| |+- UART0 transmit interrupt cause select bit

// || +- UART1 transmit interrupt cause select bit

// || 0: Transmit buffer empty (TI=1)

// || 1: Transmission completed (TXEPT=1)

// |+------- Must be fixed to '0'

// +-------- Separate /CTS//RTS bit

// 0: /CTS//RTS shared pin

// 1: /CTS//RTS separate pin

// Setting UART1 transmit/receive control register 1 (UART mode)

u1c1 = 0x04; // ---- X1X1

// |||+- Transmit enable bit

// ||| 0: Transmission disabled

// ||| 1: Transmission enabled

//	||+-- Transmit buffer empty flag

//	||	0: Data present in transmit buffer register

//	||	1: No data present in transmit buffer register

// |+--- Receive enable bit

// | 0: Reception disabled

// | 1: Reception enabled

// +---- Receive complete flag

// 0: No data present in receive buffer

// 1: Data present in receive buffer

// Setting UART1 receive interrupt control regiser. Note highest priority.

s1ric = 0x03; // ---- XXXX

// ||||

// |||+-- Interupt priority level select bit

// ||+--- Interupt priority level select bit

// |+---- Interupt priority level select bit

// | 000: Level 0 (interrupt disabled)

// | 001: Level 1

// | 010: Level 2

// | 011: Level 3

// | 100: Level 4

// | 101: Level 5

// | 110: Level 6

// | 111: Level 7

// +----- Interupt request bit

// 0: Interrupt not requested

// 1: Interrupt requested

// Setting UART1 bit rate generator for 9600 baud (UART mode)

u1brg = 0x67; // REMARKS: U1BRG=(Xin/(16*clock_select*Baud))-1

// For example:

// Xin = 16MHz

// clock_select = 1 (source=f1)

// Baud = 9600 Baud rate

// => u1brg = 103d = 0x67 (actual baud = 9615)

}

“contacts.c”

Contact sensor code for the M16C/62 development board. Untested.

#include <stdio.h>

#include "sfr62.h"

/*

Sensor 1 - left hand palm.

Sensor 2 - right hand palm.

Sensor 3 - left hand index finger.

Sensor 4 - left had middle finger.

Sensor 5 - right hand index finger.

Sensor 6 - right hand middle finger.

*/

unsigned char sensor_map[6]; /* For this test, only six contact points are present.*/

unsigned char bTransmit; /* Used to determine if data transmission should occur after a full sensor sweep.*/

#pragma INTERRUPT timer_interrupt

void main(void)

{

/* Initialise pins.*/

/*

By default, use port2 for output, port 3 for intput.

These ports are chosen simply because they are not attached

to specific periphals. Port10 is used for ADC, and port9 is

dedicated to input only.

For this test, only 4 input pins are required, and 2 output pins.

Bitwise 0 for input, 1 for output.

*/

pd2 = 0x03; /* Pins 0 and 1 of port2 are used for output.*/

pd3 = 0x00; /* Pins 0, 1, 2 and 3 or port3 are used initially for input (note: 0x00 sets all of port3 to input here).*/

/*

These circuit requirements are that the circuit goes low when not connected, hence a pull-down

resistor is required. In turn, disable pull-up resistors.

*/

pu04 = 0; /* port2, pins 0-3.*/

pu05 = 0; /* port2, pins 4-7.*/

pu06 = 0; /* port3, pins 0-3.*/

pu07 = 0; /* port3, pins 4-7.*/

/* Set outputs to low initially.*/

p2_0 = 0;

p2_1 = 0;

/* UART configuration.*/

u1mr = 0x05; /* Set 8bit mode data transfer, internal clock, one stop bit, parity bit disabled, sleep mode deselected.*/

u1c0 = 0x10; /* Further testing may show 0x00 to be the better option here.*/

ucon = 0x00;

u1brg = 0x67; /* Set 9600 Baud rate (actually 9615).*/

u1c1 = 0x01; /* Enable data transmission.*/

/*s1ric = 0x03;*/ /* Note that receiving is not really used here, so receive interrupt is left alone.*/

/* Timered interrupt configuration.*/

ta0mr = 0x80; /* 2us count source period selected.*/

ta0 = 0xC350; /* 50000. 2us * 50000 = 0.1ms, the desired timing of the overall system.*/

ta0ic = 0x01; /* Low priority set for timer A 0 interrupt (it's the only one).*/

bTransmit = 0;

while(1)

{

/*

The timer is stopped to allow a full sensor sweep before anything is transmitted.

This also garuntees the sensor data is not in a state of flux while transmitting

(it an interrupt is used for transmit, it will stop normal code execution while

in the interrupt service routine).

Stop timer.

Get sensor readings.

Restart timer.

*/

read_cycle_one(); /* First circuit sensor sweep.*/

debounce(); /* Allows the chip to switch I/O registers.*/

read_cycle_two(); /* Second circuit sensor sweep.*/

debounce(); /* Allows the chip to switch I/O registers.*/

if(bTransmit)

transmit_connections(); /* If data should be transmitted, send it.*/

bTransmit = 0; /* Data has definitely been sent by this point.*/

}

}

/* A simple timer function to debounce the circuit.*/

void debounce(void)

{

int i;

for(i = 0; i < 1000; i++)

{

}

}

/* Set a sensor contact point based upon port readings taken after a circuit on operation.*/

void set_sensors(unsigned char sensor_num, unsigned char port_reading)

{

if(port_reading & 0x01)

sensor_map[2] = sensor_num; /* Left hand index finger.*/

if(port_reading & 0x02)

sensor_map[3] = sensor_num; /* Left hand middle finger.*/

if(port_reading & 0x04)

sensor_map[4] = sensor_num; /* Right hand index finger.*/

if(port_reading & 0x08)

sensor_map[5] = sensor_num; /* Right hand index finger.*/

if(port_reading & 0x10)

sensor_map[1] = sensor_num; /* Right hand palm.*/

}

/* First cycle of reading.*/

void read_cycle_one(void)

{

unsigned char port_reading; /* What sensors are written high by turning on an output.*/

/*

It may actually use less memory space if a simple for loop were used to reset the sensors instead.

These types of optmisations should be looked at, especially if the compiled code for a memset is

no different than a simple for loop with the hardware instructions available.

*/

memset(sensor_map, 0, 6*sizeof(CONTACT_STORAGE)); /* Start each reading from scratch.*/

p2_0 = 1; /* Set palm of first glove to high.*/

debounce(); /* Allow the circuit to stabalise.*/

port_reading = p3; /* See which inputs are active as a result.*/

set_sensors(1, port_reading);

p2_0 = 0; /* Turn off the output.*/

p2_1 = 1; /* Set palm of other glove to high.*/

debounce(); /* Allow the circuit to stabalise.*/

port_reading = p3; /* Take another reading.*/

set_sensors(2, port_reading);

p2_1 = 0; /* And set the pin to low again.*/

pd2 = 0x01; /* Left glove still output, right glove palm changed to input.*/

pd3 = 0x0C; /* 0000 1100 - right glove fingers changed to output.*/

}

/* Second cycle of reading.*/

void read_cycle_two(void)

{

p3_2 = 1; /* Right hand index finger.*/

debounce();

port_reading = p3 & 0x03; /* Disregard pins that are in the output state.*/

set_sensors(5, port_reading);

p3_2 = 0;

p3_3 = 1; /* Right hand middle finger.*/

debounce();

port_reading = p3 & 0x03; /* Disregard pins that are in the output state.*/

set_sensors(6, port_reading);

p3_3 = 0;

p2_0 = 1; /* Left hand palm.*/

debounce();

port_reading = (p2 & 0x02) ? 0x10 : 0x00; /* Only checking the right palm here.*/

set_sensors(1, port_reading);

p2_0 = 0;

/* Set pins back to original I/O status.*/

pd2 = 0x03; /* Pins 0 and 1 of port2 are used for output.*/

pd3 = 0x00; /* Pins 0, 1, 2 and 3 or port3 are used initially for input (note: 0x00 sets all of port3 to input here).*/

}

void transmit_connections(void)

{

unsigned char count; /* Generic loop counter. The chip is 16bit, but the counter can be used as a sensor id.*/

/*u1c1 = 0x01;*/ /* Transmission should already be enabled.*/

/* u1tbl stores the data to be sent, and sends it when the transmit buffer is emptied.*/

for(count = 0; count < 6; count++)

{

u1tbl = count;

while(!(u1c1 & 0x02)); /* Wait for transmit buffer to empty.*/

u1tbl = sensor_map[count]; /* Send each sensor data one at a time.*/

while(!(u1c1 & 0x02)); /* Wait for transmit buffer to empty.*/

}

}

void timer_interrupt(void)

{

bTransmit = 1; /* The contact data should be transmitted ASAP.*/

}

89

lxxxix

2004-08-14T06:30:16