Re: In-memory database

Anton Ertl <anton@mips.complang.tuwien.ac.at> wrote:

Thomas Koenig <tkoenig@netcologne.de> writes:

Anton Ertl <anton@mips.complang.tuwien.ac.at> schrieb:
>
[in-memory database]
>

but the question is if
the machine has enough RAM for the database.  Our dual-Xeon system
from IIRC 2007 has 24GB of RAM, not sure how big it could be
configured; OTOH, we have a single-Xeon system from 2009 or so with
32GB of RAM (and there were bigger Xeons in the market at the time).

>
The minimum requirement of SAP HANA is 64 GB of memory, but typical
ranges are from 256GB to 1TB.

 
What is the relevance of SAP HANA for the topic at hand?
 
The question was if the RAM can hold the data.  For each account they
would have to keep the current balance (64 bits should be enough for
that), the account number (64 bits for the up to 19 digits of a Visa
card) for verifying that we are at the correct entry in the hash table
and probably some account status information (64 bits should be
plenty?).
 
There is also the sequence of transactions (a 64-bit transaction
offset in the log per transaction should be enough for that).  The
sequence of transactions may be useful for fraud detection, but I
don't know enough about that to know how to scale the system, so I'll
just say that fraud detection is done by a bigger system before the
transaction goes through to the transaction processing computer.
 
The sequence of transactions is also needed for generating the reports
and for dealing with customer complaints, but again, that's not
processing the transactions themselves (and is basically read-only,
except that the customer-complaint processing may result in additional
transactions).
 
So, with 24 bytes needed for each account on the
transaction-processing server, 32GB with, say 8GB left for
copy-on-write and other administrative purposes should be good for
about 900M accounts at a hash table load factor of 84%.  I guess that
Visa has more accounts, so one would need a box with more RAM.
 
A single core of the Xeon should easily be able to handle all the 56K
transactions per second, both the logging and the update of the hash
table, and in that case no locking is needed.  But that first needs a
sequence of transactions coming in.

AFAICS main transaction processor does not need to know about individual
cards.  Cards are issued by banks and clearly bank needs info about
card and customer.  Main transaction processor could deal only with
banks, namely verify that bank is solvent (_bank_ balance stays within
agreed limits) and that message data is legit.

OTOH, information about specific card is likely to be bigger.  Card
may have daily limit on transactions, that is another number to
keep (actually limit and daily amount of transactions).  There is
information used to verify validity of transactoion like customer
name, validity date of the card and validation code.  Financial
institutions are regulated and may be legaly obliged to keep and
check some information which is technically not needed for processing
of transactions (IIUC in my country banks are not allowed to
directly transfer money between themselfs, all transfers are
supposed to go trough central national bank).

I think that transaction center wants to keep more information
than a single copy of data in the RAM: with single copy any
memory corruption could mean loss of hours of transaction data
which is equivalent to quite a lot of cash.  So I suspect that
that there are layers of redundancy buit-in.  And even if
performencewise what they do is suboptimal they are probably very
reluctant to changes in core accounting code.

--
                              Waldek Hebisch