In the discussion of Turing Machines the concept of the tape was introduced. Computers use memory, but the idea is the same. Memory is divided into bytes, just at the tape was divided into cells. Each byte can contain one of 256 (0..255) values. The CPU can access data stored in each byte of memory individually. Byte can also be combined together to represent more values. With 2 bytes 65536 values can be represented, 4 bytes can represent 4294967296 values. Memory is used to store almost all the data that the computer uses. The instructions that comprise a program a loaded into memory, the data used by programs is stored in memory.
There are two main types of memory in a computer, random access memory [RAM] and hard drives. These two types of memory are usually used for different tasks. Hard drives are able to store data while a computer is turned off and they can store huge amounts of data. RAM, on the other hard, is thousands of times faster than hard drives. Therefore it is much faster for the CPU to access data from RAM than it is to access it from the hard drive (much like a cache).
Most modern operating systems are capable of using disk space like RAM. They copy data being used by a program to disk to allow the RAM to be used. Disk space use in this was is called virtual memory. This can be very useful if a process has allocated lots of memory, but is only using some of it (spacial locality). The operating system can then copy the unused data to disk and allow the RAM to be used by another program, or even the original program. However accessing data from the disk is very slow, so when the program wants to access data that has be copied to disk (also called swapped out) it can reduce the performance of the program greatly.
Getting more RAM for a computer can greatly increase performance if swapping is a problem. However if swapping is not a problem, adding more memory may not provide much gain in performance. However computers can always use more memory. Recently used data is often kept in memory in case it will be used again (temporal locality). One may notice this effect when running a program. Starting it the first time takes much longer than running it subsequent times. However if you wait too long between runs it will be slow again, as the memory that was being used to cache the program was used for something else.
One major difference between RAM and hard disks is that data is only stored in RAM while the power is on. Once the power is off, all data stored in memory is lost.
The amount of RAM that a computer has is usually measured in megabytes. A megabyte is 2^20 bytes. Although now, some computers are coming with gigabytes (2^30 bytes) of memory.