====================================
The client cache file implementation
====================================

This test exercises the FileCache implementation which is responsible for
maintaining the ZEO client cache on disk. Specifics of persistent cache files
are not tested.

As the FileCache calls back to the client cache we'll use a dummy to monitor
those calls:

  >>> from ZEO.tests.test_cache import ClientCacheDummy, oid
  >>> tid = oid
  >>> cache_dummy = ClientCacheDummy()

We'll instanciate a FileCache with 200 bytes of space:

  >>> from ZEO.cache import FileCache
  >>> fc = FileCache(maxsize=200, fpath=None, parent=cache_dummy)

Initially the cache is empty:

  >>> len(fc)
  0
  >>> list(fc)
  []
  >>> fc.getStats()
  (0, 0, 0, 0, 0)


Basic usage
===========

Objects are represented in the cache using a special `Object` object. Let's
start with an object of the size 100 bytes:

  >>> from ZEO.cache import Object
  >>> obj1_1 = Object(key=(oid(1), tid(1)), data='#'*100,
  ...                 start_tid=tid(1), end_tid=None)

Notice that the actual object size is a bit larger because of the headers that
are written for each object:

  >>> obj1_1.size
  120

Initially the object is not in the cache:

  >>> (oid(1), tid(1)) in fc
  False

We can add it to the cache:

  >>> fc.add(obj1_1)
  True

And now it's in the cache:

  >>> (oid(1), tid(1)) in fc
  True
  >>> len(fc)
  1

We can get it back and the object will be equal but not identical to the one we
stored:

  >>> obj1_1_copy = fc.access((oid(1), tid(1)))
  >>> obj1_1_copy.data == obj1_1.data
  True
  >>> obj1_1_copy.key == obj1_1.key
  True
  >>> obj1_1_copy is obj1_1
  False

The cache allows us to iterate over all entries in it:

  >>> list(fc)  # doctest: +ELLIPSIS
  [<ZEO.cache.Entry object at 0x...>]


When an object gets superseded we can update it. This only modifies the header,
not the actual data. This is useful when invalidations tell us about the
`end_tid` of an object:

  >>> obj1_1.data = '.' * 100
  >>> obj1_1.end_tid = tid(2)
  >>> fc.update(obj1_1)

When loading it again we can see that the data was not changed:

  >>> obj1_1_copy = fc.access((oid(1), tid(1)))
  >>> obj1_1_copy.data    # doctest: +ELLIPSIS
  '#############...################'
  >>> obj1_1_copy.end_tid
  '\x00\x00\x00\x00\x00\x00\x00\x02'

Objects can be explicitly removed from the cache:

  >>> fc.remove((oid(1), tid(1)))
  >>> len(fc)
  0
  >>> (oid(1), tid(1)) in fc
  False

Evicting objects
================

When the cached data consumes the whole cache file and more objects need to be
stored the oldest stored objects are evicted until enough space is available.
In the next sections we'll exercise some of the special cases of the file
format and look at the cache after each step.


The current state is a cache with two records: the one object which we removed
from the cache and another free record the reaches to the end of the file.

The first record has a size of 141 bytes:

  141 = 1 ('f') + 4 (size) + 8 (OID) + 8 (TID) + 8 (end_tid) +
        4 (data length) + 100 (old data) + 8 (OID)

The second record has a size of 47 bytes:

  47 = 1 ('f') + 8 (size) + 38 (free space)

Note that the last byte is an 'x' because the initialisation of the cache file
forced the absolute size of the file by seeking to byte 200 and writing an 'x'.

  >>> from ZEO.tests.test_cache import hexprint
  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 66 00 00 00  |ZEC4........f...|
  00000010  00 00 00 00 8d 00 00 00  01 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  02 00 00 00 64 23 23 23  |............d###|
  00000030  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000040  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000050  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000060  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000070  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000080  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000090  23 00 00 00 00 00 00 00  01 66 00 00 00 00 00 00  |#........f......|
  000000a0  00 2f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |./..............|
  000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
  000000c0  00 00 00 00 00 00 00 78                           |.......x        |

Case 1: Allocating a new block that fits after the last used one

  >>> obj2_1 = Object(key=(oid(2), tid(1)), data='******',
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj2_1)
  True

The new block fits exactly in the remaining 47 bytes (41 bytes header + 6
bytes payload) so the beginning of the data is the same except for the last 47
bytes:

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 66 00 00 00  |ZEC4........f...|
  00000010  00 00 00 00 8d 00 00 00  01 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  02 00 00 00 64 23 23 23  |............d###|
  00000030  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000040  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000050  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000060  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000070  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000080  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
  00000090  23 00 00 00 00 00 00 00  01 61 00 00 00 2f 00 00  |#........a.../..|
  000000a0  00 00 00 00 00 02 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 06 2a 2a 2a 2a 2a 2a  |..........******|
  000000c0  00 00 00 00 00 00 00 02                           |........        |

Case 2: Allocating a block that wraps around and frees *exactly* one block

  >>> obj3_1 = Object(key=(oid(3), tid(1)), data='@'*100,
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj3_1)
  True

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 61 00 00 00  |ZEC4........a...|
  00000010  8d 00 00 00 00 00 00 00  03 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  00 00 00 00 64 40 40 40  |............d@@@|
  00000030  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000040  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000050  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000060  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000070  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000080  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000090  40 00 00 00 00 00 00 00  03 61 00 00 00 2f 00 00  |@........a.../..|
  000000a0  00 00 00 00 00 02 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 06 2a 2a 2a 2a 2a 2a  |..........******|
  000000c0  00 00 00 00 00 00 00 02                           |........        |

Case 3: Allocating a block that requires 1 byte less than the next block

  >>> obj4_1 = Object(key=(oid(4), tid(1)), data='~~~~~',
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj4_1)
  True

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 61 00 00 00  |ZEC4........a...|
  00000010  8d 00 00 00 00 00 00 00  03 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  00 00 00 00 64 40 40 40  |............d@@@|
  00000030  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000040  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000050  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000060  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000070  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000080  40 40 40 40 40 40 40 40  40 40 40 40 40 40 40 40  |@@@@@@@@@@@@@@@@|
  00000090  40 00 00 00 00 00 00 00  03 61 00 00 00 2e 00 00  |@........a......|
  000000a0  00 00 00 00 00 04 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 05 7e 7e 7e 7e 7e 00  |..........~~~~~.|
  000000c0  00 00 00 00 00 00 04 31                           |.......1        |

Case 4: Allocating a block that requires 2 bytes less than the next block

  >>> obj4_1 = Object(key=(oid(5), tid(1)), data='^'*98,
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj4_1)
  True

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 61 00 00 00  |ZEC4........a...|
  00000010  8b 00 00 00 00 00 00 00  05 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  00 00 00 00 62 5e 5e 5e  |............b^^^|
  00000030  5e 5e 5e 5e 5e 5e 5e 5e  5e 5e 5e 5e 5e 5e 5e 5e  |^^^^^^^^^^^^^^^^|
  00000040  5e 5e 5e 5e 5e 5e 5e 5e  5e 5e 5e 5e 5e 5e 5e 5e  |^^^^^^^^^^^^^^^^|
  00000050  5e 5e 5e 5e 5e 5e 5e 5e  5e 5e 5e 5e 5e 5e 5e 5e  |^^^^^^^^^^^^^^^^|
  00000060  5e 5e 5e 5e 5e 5e 5e 5e  5e 5e 5e 5e 5e 5e 5e 5e  |^^^^^^^^^^^^^^^^|
  00000070  5e 5e 5e 5e 5e 5e 5e 5e  5e 5e 5e 5e 5e 5e 5e 5e  |^^^^^^^^^^^^^^^^|
  00000080  5e 5e 5e 5e 5e 5e 5e 5e  5e 5e 5e 5e 5e 5e 5e 00  |^^^^^^^^^^^^^^^.|
  00000090  00 00 00 00 00 00 05 32  03 61 00 00 00 2e 00 00  |.......2.a......|
  000000a0  00 00 00 00 00 04 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 05 7e 7e 7e 7e 7e 00  |..........~~~~~.|
  000000c0  00 00 00 00 00 00 04 31                           |.......1        |

Case 5: Allocating a block that requires 3 bytes less than the next block

The end of the file is already a bit crowded and would create a rather complex
situation to work on. We create an entry with the size of 95 byte which will
be inserted at the beginning of the file, leaving a 3 byte free space after
it.

  >>> obj4_1 = Object(key=(oid(6), tid(1)), data='+'*95,
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj4_1)
  True

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 61 00 00 00  |ZEC4........a...|
  00000010  88 00 00 00 00 00 00 00  06 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  00 00 00 00 5f 2b 2b 2b  |............_+++|
  00000030  2b 2b 2b 2b 2b 2b 2b 2b  2b 2b 2b 2b 2b 2b 2b 2b  |++++++++++++++++|
  00000040  2b 2b 2b 2b 2b 2b 2b 2b  2b 2b 2b 2b 2b 2b 2b 2b  |++++++++++++++++|
  00000050  2b 2b 2b 2b 2b 2b 2b 2b  2b 2b 2b 2b 2b 2b 2b 2b  |++++++++++++++++|
  00000060  2b 2b 2b 2b 2b 2b 2b 2b  2b 2b 2b 2b 2b 2b 2b 2b  |++++++++++++++++|
  00000070  2b 2b 2b 2b 2b 2b 2b 2b  2b 2b 2b 2b 2b 2b 2b 2b  |++++++++++++++++|
  00000080  2b 2b 2b 2b 2b 2b 2b 2b  2b 2b 2b 2b 00 00 00 00  |++++++++++++....|
  00000090  00 00 00 06 33 00 05 32  03 61 00 00 00 2e 00 00  |....3..2.a......|
  000000a0  00 00 00 00 00 04 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 05 7e 7e 7e 7e 7e 00  |..........~~~~~.|
  000000c0  00 00 00 00 00 00 04 31                           |.......1        |

Case 6: Allocating a block that requires 6 bytes less than the next block

As in our previous case, we'll write a block that only fits in the first
block's place to avoid dealing with the cluttering at the end of the cache
file.

  >>> obj4_1 = Object(key=(oid(7), tid(1)), data='-'*89,
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj4_1)
  True

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 61 00 00 00  |ZEC4........a...|
  00000010  82 00 00 00 00 00 00 00  07 00 00 00 00 00 00 00  |................|
  00000020  01 00 00 00 00 00 00 00  00 00 00 00 59 2d 2d 2d  |............Y---|
  00000030  2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  |----------------|
  00000040  2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  |----------------|
  00000050  2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  |----------------|
  00000060  2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  |----------------|
  00000070  2d 2d 2d 2d 2d 2d 2d 2d  2d 2d 2d 2d 2d 2d 2d 2d  |----------------|
  00000080  2d 2d 2d 2d 2d 2d 00 00  00 00 00 00 00 07 36 00  |------........6.|
  00000090  00 00 00 06 33 00 05 32  03 61 00 00 00 2e 00 00  |....3..2.a......|
  000000a0  00 00 00 00 00 04 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 05 7e 7e 7e 7e 7e 00  |..........~~~~~.|
  000000c0  00 00 00 00 00 00 04 31                           |.......1        |

Case 7: Allocating a block that requires >= 5 bytes less than the next block

Again, we replace the block at the beginning of the cache.

  >>> obj4_1 = Object(key=(oid(8), tid(1)), data='='*80,
  ...                 start_tid=tid(1), end_tid=None)
  >>> fc.add(obj4_1)
  True

  >>> hexprint(fc.f)
  00000000  5a 45 43 34 00 00 00 00  00 00 00 00 61 00 00 00  |ZEC4........a...|
  00000010  79 00 00 00 00 00 00 00  08 00 00 00 00 00 00 00  |y...............|
  00000020  01 00 00 00 00 00 00 00  00 00 00 00 50 3d 3d 3d  |............P===|
  00000030  3d 3d 3d 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 3d 3d 3d  |================|
  00000040  3d 3d 3d 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 3d 3d 3d  |================|
  00000050  3d 3d 3d 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 3d 3d 3d  |================|
  00000060  3d 3d 3d 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 3d 3d 3d  |================|
  00000070  3d 3d 3d 3d 3d 3d 3d 3d  3d 3d 3d 3d 3d 00 00 00  |=============...|
  00000080  00 00 00 00 08 66 00 00  00 00 00 00 00 09 36 00  |.....f........6.|
  00000090  00 00 00 06 33 00 05 32  03 61 00 00 00 2e 00 00  |....3..2.a......|
  000000a0  00 00 00 00 00 04 00 00  00 00 00 00 00 01 00 00  |................|
  000000b0  00 00 00 00 00 00 00 00  00 05 7e 7e 7e 7e 7e 00  |..........~~~~~.|
  000000c0  00 00 00 00 00 00 04 31                           |.......1        |

Statistic functions
===================

The `getStats` method talks about the added objects, added bytes, evicted
objects, evicted bytes and accesses to the cache:

  >>> fc.getStats()
  (8, 901, 5, 593, 2)

We can reset the stats by calling the `clearStats` method:

  >>> fc.clearStats()
  >>> fc.getStats()
  (0, 0, 0, 0, 0)


Cleanup
=======

As the cache is non-persistent, its file will be gone from disk after closing
the cache:

  >>> fc.f  # doctest: +ELLIPSIS
  <open file '<fdopen>', mode 'w+b' at 0x...>
  >>> fc.close()
  >>> fc.f
