MULTIMEDIA

What is Multimedia?

In the age of the desktop computer multimedia has come to mean the integration of two or more different media - audio, video, animation, graphics, text - within the computer.

Multimedia represents a qualitative technological and conceptual advance in information technology. Sound and moving images can convey vastly greater quantities of information than simple text and graphics will greatly change not only the way in which that information is transmitted and used but also how it is structured and stored. This new model of information is called hypermedia.

Hypermedia

One of the problems of early computer systems was that they were very sequential and linear, and they forced their users into the same mold. Text, in particular, was presented in a page-by-page mode - if the user wanted to get to a certain topic in a large text file such as an on-line manual they often had to page down through screen upon screen to reach the desired location.

It was soon realised - not least by the poor users themselves! - that this was not a very efficient way for humans to assimilate information, so in the mid-sixties Theodore Nelson created the concept of hypertext. A hypertext system consists of nodes - which contain the text - and links between the nodes, which define the paths the user can follow to access the text in non-sequential ways. The links represent associations of meaning and can be thought of as cross-references. This structure is created by the author of the system, although in more sophisticated hypertext systems the user is able to define their own paths .

Hypermedia represents an evolution of the hypertext concept. In hypermedia systems the nodes within the web may each contain one or more types of data, from simple text to audio and video clips. Hypermedia nodes are coming to be known by the confusing term document, which represents a collection of data comprising different media - text with a video clip, for example.
Hypertext and hypermedia have the advantage over sequentially-presented information by more closely modelling the human memory process. We learn by association, by attaching meaning to information and creating our own nodes and links within our brains . We don't store information in series of bits, as do computers.
These hypersystems also present data in a more human-friendly way. For example, our visual system has evolved to be sensitive to movement in our environment, so video and animation represent much more natural ways of communication than do text and still images.
Hypermedia enables a change in our relationship with the computer. Instead of being shoehorned into the closed and sequential world of the computer we can use the computer as an extension of ourselves to augment our existing capabilities. Instead of the user being a passive component of the computer system the system becomes a tool for the active user, a more open concept the potential of which is limited only by technological constraints and the imagination of users.

Elements of Multimedia

There is no exhaustive list of the data types that can be included in a multimedia document - any medium that can be represented digitally in the computer can be used in multimedia. When virtual reality tools such as data gloves which translate the hand movements of the wearer into computer instructions, become commonplace they will become a part of multimedia. It's not hard to envisage a document about an object which would allow the user to 'feel' the object. However, at the time of writing it's generally accepted that a multimedia document contains audio and/or moving video data in addition to any text or still images.

Sound

Digital audio is far from new - a glance inside a record shop to see the preponderance of CDs over vinyl LPs confirms that digital sound is a mature and well-established technology. Sound is converted from its natural wave form to digital data via the process of sampling. The sound wave - in electronic form - is fed into a specialised ADC (Analogue to Digital Converter) chip which measures the wave level a number of times per second as determined by the sampling frequency (measured in kHz). The higher the sampling frequency the less information is discarded from the original signal and the greater is the sound fidelity. The digital sound is played back using a DAC (Digital to Analogue Converter).
The other determinant of sound quality is the accuracy of the sample value, measured in bits; at the time of writing the highest quality digital audio is16-bit. The main cost to be paid for quality is, of course, memory. High quality CD-Audio sound - 16-bit/44.1 kHz stereo - uses 170 kB/sec. However, sound of such quality is seldom used in multimedia documents, where 8-bit/11 kHz voice-quality mono sound at 10 kB/sec is usually sufficient.
Digital sound can also be generated using the MIDI (Musical Instrument Digital Interface) protocol. Unlike files of digitised sound waves a MIDI file does not contain the sound itself - rather it comprises a set of instructions to a MIDI device such as a synthesizer or a computer. Each instruction tells the device what note to play, in what 'voice' (type of sound, such as a piano), and for how long. These commands are stored in a small number of bytes so MIDI files are a lot smaller than files containing the sound data, but of course the generated sounds are much less complex than digitised sounds. This is analogous to the difference between vector and bitmap graphics. The small size of MIDI data means that electronic music may be incorporated into documents at little cost in terms of file size.
Digital audio is an integral part of Apple Macs, all of which have an internal speaker capable of playing good quality sound, and some of which are capable of recording sound direct from an analogue source such as a microphone. PCs were never designed with audio in mind and as such only have a poor quality internal speaker which can only be used for emitting the occasional beep. It is, however, possible to purchase a sound card quite cheaply to go into an expansion slot inside the PC which will produce high quality audio when used with ordinary speakers.

Animation

Animation consists of still images - frames - displayed so quickly that they give the impression of continuous movement. Although this appears to be a simple task, in fact it requires a lot of number-crunching and places great demands on the computer.
The reason for this is that nearly all of the animation is mathematically-defined. Each screen object - actor - is a vector image. The movement of actors along paths is calculated using numerical transformations applied to their defining coordinates, and if these transformations have to be carried out on floating-point rather than integer values, as will often be the case, then calculation time will be considerably slower. Moreover, to give the impression of smoothness the frame rate has to be at least 16 frames per second (fps), and for natural-looking motion 25 fps is required. The combination of these factors produces a vast arithmetical workload, which until recently was beyond the power of desktop microcomputers. However, the rapid and almost exponential increase in the processing power of micros has enabled run-of-the-mill PCs and Macs to create and run animations, although a maths coprocessor - a chip dedicated to maths calculations which resides on the motherboard - is a highly desirable accessory.
Three-dimensional animation - where, for example, the viewer might appear to 'fly' around an object in order to see it from different perspectives - is an order of magnitude more complicated than 2-D animation. In addition to the extra complexity added to actors and their paths through the scene various other factors have to be taken into account, such as the lighting of the scene, the casting of shadows, the use of perspective and the partial obscuring of one object by another in front of it. These factors generate such an immense number of calculations that 3-D animations are still restricted to heavy-duty graphics workstations.
In relation to digital video animation uses very little memory as it consists mainly of drawing and moving instructions, and as such it's very useful for multimedia applications where moving visuals are required to make a point but where digital video may be unsuitable, unnecessary, or too expensive in terms of disk space or memory.

Video

Video consists of photographic images that are played back at speeds of 15-30 frames per second and provide the appearance of full motion. To use video in a multimedia application, the developer has to capture, digitize and edit the video segments using special video production hardware and software. Video can also be captured directly in digital format using a digital video camera. Due to the size of video files, incorporating video into a multimedia application is often a challenge. Files require large amounts of storage space, therefore they are often compressed. Video compression works by recognizing that only a small portion of the video image changes from frame to frame, and after storing the first reference frame only changes from one frame to the next are stored. The Motion Pictures Experts Group has defined a standard for video and audio compression and de-compression, called MPEG. MPEG compression can reduce the size of video files up to 95 percent, while retaining near TV quality. Video compression has allowed video to play a more important role in multimedia applications. Technologies such as streaming video made video a viable part of multimedia on the Web. As with streaming audio, streaming video allows the user to view longer or live video images as they are downloaded to the computer. The standard used for transmitting video data on the Internet is RealVideo, which is a component of RealPlayer supported by most current Web browsers. Streaming video also allows conducting Internet videoconferences that work like Internet telephony. A video camera, videoconferencing software and video capture card digitize and compress the images and sounds. After they are sent over the Internet, equipment and software at the receiving end assemble and decompress the data presenting the images and sound as video. The Synchronized Multimedia Integration Language (SMIL, pronounced "smile") enables simple authoring of interactive audiovisual presentations. SMIL is typically used for "rich media/multimedia" presentations, which integrate streaming audio and video with other media type. SMIL is an HTML-like language and many SMIL presentations are written using a simple text-editor.

Hardware & Software for Digital Video

The capture and playback of digital video involves the moving of massive volumes of data around the computer system at high speed, and as such places enormous demands on the hardware. In the most extreme case, one second of full-screen, 25 fps, 24-bit colour video - the equivalent of a normal TV picture - uses 26 MB of data. Apart from the obvious disk storage and RAM problems a data transfer rate of 26 MB/s is well beyond the current capabilities of even the best microcomputers and networks. Unsurprisingly, full-screen full-motion video (FSFMV) remains an unattainable ideal in the short- to medium-term future. Today's digital video runs in a window on screen, and the frame rate and colour depth vary from application to application.

The development of applications containing digital video requires fast machines and specialised hardware in the form of plug-in cards for video capture and compression. Playback is best accomplished on computers that contain the same cards. However, it is now the case that the applications can be run on ordinary micros without any additional hardware, the video being decoded and played back in software alone. At the time of writing this is accomplished by Apple Quicktime on the Mac and Microsoft Video for Windows on the PC, both of which enable the playback of digital video in small screen windows at fairly low frame rates, although the window size and frame rate available will undoubtedly increase with time as micros become more powerful.

Nevertheless, even a short video clip in a small window takes up a lot of RAM and disk space. For example, a Video for Windows clip playing in a 160 x 120 pixel window at 15 fps and 256 colours comprises 160 x 120 x 15 = 288,000 bytes (281 kB) per second of raw video data, not including audio. Of course, this data can be compressed by means of sophisticated techniques such as frame differencing - where only the differences between one frame and the next are saved, rather than all the frame data - but the sad fact remains that video files will always be measured in megabytes, making CDROM the only feasible distribution medium for applications incorporating digital video.

Applications of Multimedia and Hypermedia

Multimedia is much closer to the way humans naturally communicate and gather information about the world than the traditional text-based computing paradigm that preceded it, and hypermedia in particular is much more akin to the associative, semantic way we learn than linear sequential material. Not unsurprisingly, then, the main application areas for hypermedia have been those involved in learning: education, training and reference works.

Hypertext and hypermedia systems are passive, insofar as they wait for the user to act upon them rather than forcing the user to take actions. This is a necessary feature of their non-linearity which allows users to find information by navigating from node to node in unpredictable sequences, and these qualities of passivity and non-linearity make hypersystems ideal for browsing-type applications. CDROM Encyclopedias, such as Microsoft's Encarta or McGraw-Hill's Multimedia Encyclopedia of Mammalian Biology, are educational hypermedia reference works in which the user can navigate from topic to topic, or cross-reference information, in ways which would not be possible with traditional volumes. Moreover the works use graphics, sound, animation and digital video to great effect.

Language learning is already an analogue multimedia activity, with both audio and video tapes used extensively, and plainly this is an area which could benefit from the application of digital multimedia. A number of CDROM-based CALL (Computer-Aided Language Learning) applications utilising digital audio are already on the market and laserdisks are quite popular in the field, so digital video cannot be far away.

Online information systems, such as technical references and manuals, are excellent applications of hypermedia: not only can users find the information they want when they want it, but by doing so their retention is better than if they had simply read through a manual. Moreover, a manual won't contain a video clip illustrating a procedure, or an audio sequence listing the diagnostic sounds that a component makes, or an animation showing a piece of equipment rotating in three dimensions.

Multimedia can be of considerable benefit to disabled users. Traditionally anyone unable to use a keyboard or see a screen was necessarily unable to use computers. However, when computer and user can communicate via such a variety of media there is no reason, other than cost, why disabled people cannot fully participate in computing. People with visual impairment, for example, can speak to voice-activated computers to give commands and enter data, and have the computer talk to them ; similarly those physically unable to use keyboards can communicate via touch screens and pointing devices, and in the not so distant future use virtual reality datagloves which would allow the computer to communicate to them using touch.

Because of the open-ended, human-oriented nature of hypermedia it has a vast number of potential applications in nearly every area of human activity.

MULTIMEDIA ELEMENTS

STORY ELEMENTS

Perceptual-cognitive-emotional processes

Emotional outcomes

PL. CHECK FOR NEXT PART OF TUTORIAL