EvEmu  0.8.4
11 September 2021
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
MarshalStream Class Reference

Turns Python objects into marshal bytecode. More...

#include "EVEMarshal.h"

Inheritance diagram for MarshalStream:
Collaboration diagram for MarshalStream:

Public Member Functions

 MarshalStream ()
 
bool Save (const PyRep *rep, Buffer &into)
 

Protected Member Functions

bool SaveStream (const PyRep *rep)
 
template<typename T >
void Put (const T &value)
 
template<typename Iter >
void Put (Iter first, Iter last)
 
void PutSizeEx (uint32 size)
 
bool VisitInteger (const PyInt *rep)
 
bool VisitLong (const PyLong *rep)
 Adds a long to the stream. More...
 
bool VisitBoolean (const PyBool *rep)
 Adds a boolean to the stream. More...
 
bool VisitReal (const PyFloat *rep)
 Adds a double to the stream. More...
 
bool VisitNone (const PyNone *rep)
 Adds a None object to the stream. More...
 
bool VisitBuffer (const PyBuffer *rep)
 Adds a buffer to the stream. More...
 
bool VisitString (const PyString *rep)
 add a string object to the data stream More...
 
bool VisitWString (const PyWString *rep)
 add a wide string object to the data stream More...
 
bool VisitToken (const PyToken *rep)
 add a token object to the data stream More...
 
bool VisitTuple (const PyTuple *rep)
 
bool VisitList (const PyList *rep)
 
bool VisitDict (const PyDict *rep)
 
bool VisitObject (const PyObject *rep)
 Adds an object to the stream. More...
 
bool VisitObjectEx (const PyObjectEx *rep)
 Adds a New object to the stream. More...
 
bool VisitPackedRow (const PyPackedRow *pyPackedRow)
 Adds a packed row to the stream. More...
 
bool VisitSubStruct (const PySubStruct *rep)
 Adds a sub structure to the stream. More...
 
bool VisitSubStream (const PySubStream *rep)
 Adds a sub stream to the stream. More...
 
bool VisitChecksumedStream (const PyChecksumedStream *rep)
 Adds a checksumed stream to the stream. More...
 
- Protected Member Functions inherited from PyVisitor
virtual ~PyVisitor ()
 

Private Member Functions

void SaveVarInteger (const PyLong *v)
 
bool SaveRLE (const Buffer &in)
 

Private Attributes

BuffermBuffer
 

Detailed Description

Turns Python objects into marshal bytecode.

Author
Captnoord, Bloody.Rabbit

Definition at line 59 of file EVEMarshal.h.

Constructor & Destructor Documentation

MarshalStream::MarshalStream ( )

initializes object

Definition at line 64 of file EVEMarshal.cpp.

65 : mBuffer( nullptr )
66 {
67 }
Buffer * mBuffer
Definition: EVEMarshal.h:146

Member Function Documentation

template<typename T >
void MarshalStream::Put ( const T &  value)
inlineprotected

adds given value to the data stream

Definition at line 75 of file EVEMarshal.h.

References Buffer::Append(), and mBuffer.

Referenced by SaveRLE(), SaveVarInteger(), VisitBuffer(), VisitString(), VisitSubStream(), VisitToken(), and VisitWString().

75 { mBuffer->Append<T>( value ); }
void Append(const T &value)
Appends a single value to buffer.
Definition: Buffer.h:437
Buffer * mBuffer
Definition: EVEMarshal.h:146

Here is the call graph for this function:

Here is the caller graph for this function:

template<typename Iter >
void MarshalStream::Put ( Iter  first,
Iter  last 
)
inlineprotected

adds given bytes to the data stream

Definition at line 78 of file EVEMarshal.h.

References Buffer::AppendSeq(), and mBuffer.

78 { mBuffer->AppendSeq<Iter>( first, last ); }
Buffer * mBuffer
Definition: EVEMarshal.h:146
void AppendSeq(Iter first, Iter last)
Appends a sequence of elements to buffer.
Definition: Buffer.h:455

Here is the call graph for this function:

void MarshalStream::PutSizeEx ( uint32  size)
inlineprotected

utility for extended size.

Definition at line 81 of file EVEMarshal.h.

Referenced by SaveRLE(), SaveVarInteger(), VisitBuffer(), VisitDict(), VisitList(), VisitString(), VisitSubStream(), VisitToken(), VisitTuple(), and VisitWString().

82  {
83  if( size < 0xFF )
84  {
85  Put<uint8>( size );
86  }
87  else
88  {
89  Put<uint8>( 0xFF );
90  Put<uint32>( size );
91  }
92  }

Here is the caller graph for this function:

bool MarshalStream::Save ( const PyRep rep,
Buffer into 
)

saves given rep to given buffer

Definition at line 69 of file EVEMarshal.cpp.

References mBuffer, and SaveStream().

Referenced by Marshal().

70 {
71  mBuffer = &into;
72  bool res(SaveStream(rep));
73  mBuffer = nullptr;
74 
75  return res;
76 }
Buffer * mBuffer
Definition: EVEMarshal.h:146
bool SaveStream(const PyRep *rep)
Definition: EVEMarshal.cpp:78

Here is the call graph for this function:

Here is the caller graph for this function:

bool MarshalStream::SaveRLE ( const Buffer in)
private

Definition at line 524 of file EVEMarshal.cpp.

References Put(), PutSizeEx(), and Buffer::size().

Referenced by VisitPackedRow().

525 {
526  // TODO: REWRITE THIS, AS IT IS RIGHT NOW IS INEFFICIENT, I'VE CONVERTED THE BUFFER CLASS TO A BASTARDIZED VERSION OF A NORMAL BYTE ARRAY
527  // ALMAMU - 2021/04/22 - After many years the buggy "SaveZeroCompressed" function has been laid to rest
528  // may this todo be a way to remind us how unstable and fragile the EVEmu core is
529  // this code has been validated against the disassembly of Apocrypha's blue.dll
530  // for those interested, the function lives in .text:10082D10 of that DLL
531  // "hopefully" this brings our marshaller closer to fully featured
532 
533  // reserve double the buffer size just in case, we do not want to run out of space or else the iterators will start to complain
534  Buffer out(in.size() * 2, 0);
535 
536  // this code has been used and ported through different projects
537  // both ntt's reverence and Captnoord's re-implementation of evemu core have the exact same code
538  // after validation against the disassembly (and unless I've missed anything) it seems to be 100% accurate
539  // so no real modification to it was done
540  int nibble = 0;
541  int nibble_ix = 0;
542  int in_ix = 0;
543  int out_ix = 0;
544  int start, end, count;
545  int zerochains = 0;
546  int in_size = in.size();
547 
548  while(in_ix < in_size)
549  {
550  if(!nibble)
551  {
552  nibble_ix = out_ix++;
553  out[nibble_ix] = 0;
554  }
555 
556  start = in_ix;
557  end = in_ix+8;
558  if(end > in_size)
559  end = in_size;
560 
561  if(in[in_ix])
562  {
563  zerochains = 0;
564  do {
565  out[out_ix++] = in[in_ix++];
566  } while(in_ix<end && in[in_ix]);
567  count = (start - in_ix) + 8;
568  }
569  else
570  {
571  zerochains++;
572  while(in_ix<end && !in[in_ix])
573  in_ix++;
574  count = (in_ix - start) + 7;
575  }
576 
577  if(nibble)
578  out[nibble_ix] |= (count << 4);
579  else
580  out[nibble_ix] = count;
581  nibble = !nibble;
582  }
583 
584  if(nibble && zerochains)
585  zerochains++;
586 
587  while(zerochains>1)
588  {
589  zerochains -= 2;
590  out_ix -= 1;
591  }
592 
593  // Write the packed in
594  PutSizeEx( (uint32) out_ix);
595  if ( 0 < out.size() )
596  Put( out.begin<uint8>(), out.begin<uint8>() + out_ix );
597 
598  return true;
599 }
unsigned __int8 uint8
Definition: eve-compat.h:46
Generic class for buffers.
Definition: Buffer.h:40
void Put(const T &value)
Definition: EVEMarshal.h:75
unsigned __int32 uint32
Definition: eve-compat.h:50
size_type size() const
Definition: Buffer.h:610
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

Here is the caller graph for this function:

bool MarshalStream::SaveStream ( const PyRep rep)
protected

saves new stream with given rep.

Definition at line 78 of file EVEMarshal.cpp.

References MarshalHeaderByte, and PyRep::visit().

Referenced by Save().

79 {
80  Put<uint8>( MarshalHeaderByte );
81  /*
82  * Mapcount
83  * the amount of referenced objects within a marshal stream.
84  * Note: Atm not supported.
85  * (allan) have not found any information on this, so no idea how/when to implement it (or even if we need to)
86  */
87  Put<uint32>( 0 ); // Mapcount
88 
89  return rep->visit( *this );
90 }
static const uint8 MarshalHeaderByte
virtual bool visit(PyVisitor &v) const =0
Visits object.

Here is the call graph for this function:

Here is the caller graph for this function:

void MarshalStream::SaveVarInteger ( const PyLong v)
private

Definition at line 503 of file EVEMarshal.cpp.

References DoIntegerSizeCheck, Op_PyLongLong, Op_PyVarInteger, Put(), PutSizeEx(), and PyLong::value().

Referenced by VisitLong().

504 {
505  const int64 value(v->value());
506  uint8 integerSize(0);
507 
508 #define DoIntegerSizeCheck(x) if ( ( (uint8*)&value )[x] != 0 ) integerSize = x + 1;
512 #undef DoIntegerSizeCheck
513 
514  if ( integerSize > 0 && integerSize < 7 ) {
515  Put<uint8>(Op_PyVarInteger);
516  PutSizeEx(integerSize);
517  Put( &( (uint8*)&value )[0], &( (uint8*)&value )[integerSize] );
518  } else {
519  Put<uint8>(Op_PyLongLong); // 1
520  Put<int64>(value); // 8
521  }
522 }
unsigned __int8 uint8
Definition: eve-compat.h:46
void Put(const T &value)
Definition: EVEMarshal.h:75
int64 value() const
Definition: PyRep.h:278
#define DoIntegerSizeCheck(x)
signed __int64 int64
Definition: eve-compat.h:51
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

Here is the caller graph for this function:

bool MarshalStream::VisitBoolean ( const PyBool rep)
protectedvirtual

Adds a boolean to the stream.

Reimplemented from PyVisitor.

Definition at line 123 of file EVEMarshal.cpp.

References Op_PyFalse, Op_PyTrue, and PyBool::value().

124 {
125  if (rep->value())
126  Put<uint8>( Op_PyTrue );
127  else
128  Put<uint8>( Op_PyFalse );
129 
130  return true;
131 }
bool value() const
Definition: PyRep.h:340

Here is the call graph for this function:

bool MarshalStream::VisitBuffer ( const PyBuffer rep)
protectedvirtual

Adds a buffer to the stream.

Reimplemented from PyVisitor.

Definition at line 151 of file EVEMarshal.cpp.

References PyBuffer::content(), Op_PyBuffer, Put(), and PutSizeEx().

152 {
153  Put<uint8>( Op_PyBuffer );
154 
155  const Buffer& buf = rep->content();
156 
157  PutSizeEx( (uint32)buf.size() );
158  Put( buf.begin<uint8>(), buf.end<uint8>() );
159 
160  return true;
161 }
unsigned __int8 uint8
Definition: eve-compat.h:46
const Buffer & content() const
Get the const PyBuffer content.
Definition: PyRep.h:407
Generic class for buffers.
Definition: Buffer.h:40
void Put(const T &value)
Definition: EVEMarshal.h:75
unsigned __int32 uint32
Definition: eve-compat.h:50
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitChecksumedStream ( const PyChecksumedStream rep)
protectedvirtual

Adds a checksumed stream to the stream.

TODO: check the implementation of this...

Reimplemented from PyVisitor.

Definition at line 493 of file EVEMarshal.cpp.

References PyChecksumedStream::checksum(), Op_PyChecksumedStream, and PyVisitor::VisitChecksumedStream().

494 {
495  assert(false && "MarshalStream on the server side should never send checksummed objects");
496 
497  Put<uint8>(Op_PyChecksumedStream);
498 
499  Put<uint32>( rep->checksum() );
500  return PyVisitor::VisitChecksumedStream( rep );
501 }
uint32 checksum() const
Definition: PyRep.h:1082
virtual bool VisitChecksumedStream(const PyChecksumedStream *rep)
Definition: PyVisitor.cpp:133

Here is the call graph for this function:

bool MarshalStream::VisitDict ( const PyDict rep)
protectedvirtual

Add a dict object to the stream

Reimplemented from PyVisitor.

Definition at line 251 of file EVEMarshal.cpp.

References PyDict::begin(), PyDict::end(), Op_PyDict, PutSizeEx(), and PyDict::size().

252 {
253  uint32 size(rep->size());
254  Put<uint8>( Op_PyDict );
255  PutSizeEx( size );
256 
257  //we have to reverse the order of key/value to be value/key, so do not call base class.
258  PyDict::const_iterator cur = rep->begin(), end = rep->end();
259  for (; cur != end; ++cur) {
260  if ( !cur->second->visit( *this ) )
261  return false;
262  if ( !cur->first->visit( *this ) )
263  return false;
264  }
265 
266  return true;
267 }
size_t size() const
Definition: PyRep.h:769
unsigned __int32 uint32
Definition: eve-compat.h:50
const_iterator begin() const
Definition: PyRep.h:766
storage_type::const_iterator const_iterator
Definition: PyRep.h:750
const_iterator end() const
Definition: PyRep.h:767
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitInteger ( const PyInt rep)
protectedvirtual

adds a integer to the data stream

Note
assuming the value is unsigned research shows that Op_PyByte can be negative

Reimplemented from PyVisitor.

Definition at line 92 of file EVEMarshal.cpp.

References Op_PyByte, Op_PyLong, Op_PyMinusOne, Op_PyOneInteger, Op_PySignedShort, Op_PyZeroInteger, and PyInt::value().

93 {
94  const int32 val(rep->value());
95 
96  if ( val == -1 ) {
97  Put<uint8>( Op_PyMinusOne );
98  } else if ( val == 0 ) {
99  Put<uint8>( Op_PyZeroInteger );
100  } else if ( val == 1 ) {
101  Put<uint8>( Op_PyOneInteger );
102  } else if ( val + 0x8000u > 0xFFFF ) {
103  Put<uint8>( Op_PyLong );
104  Put<int32>( val );
105  } else if ( val + 0x80u > 0xFF ) {
106  Put<uint8>( Op_PySignedShort );
107  Put<int16>( val );
108  } else {
109  Put<uint8>( Op_PyByte );
110  Put<int8>( val );
111  }
112 
113  return true;
114 }
int32 value() const
Definition: PyRep.h:247
signed __int32 int32
Definition: eve-compat.h:49

Here is the call graph for this function:

bool MarshalStream::VisitList ( const PyList rep)
protectedvirtual

Add a list object to the stream

Reimplemented from PyVisitor.

Definition at line 236 of file EVEMarshal.cpp.

References Op_PyEmptyList, Op_PyList, Op_PyOneList, PutSizeEx(), PyList::size(), and PyVisitor::VisitList().

237 {
238  uint32 size(rep->size());
239  if ( size == 0 ) {
240  Put<uint8>( Op_PyEmptyList );
241  } else if ( size == 1 ) {
242  Put<uint8>( Op_PyOneList );
243  } else {
244  Put<uint8>( Op_PyList );
245  PutSizeEx( size );
246  }
247 
248  return PyVisitor::VisitList( rep );
249 }
unsigned __int32 uint32
Definition: eve-compat.h:50
size_t size() const
Definition: PyRep.h:663
virtual bool VisitList(const PyList *rep)
Definition: PyVisitor.cpp:47
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitLong ( const PyLong rep)
protectedvirtual

Adds a long to the stream.

Reimplemented from PyVisitor.

Definition at line 116 of file EVEMarshal.cpp.

References SaveVarInteger().

117 {
118  SaveVarInteger (rep);
119 
120  return true;
121 }
void SaveVarInteger(const PyLong *v)
Definition: EVEMarshal.cpp:503

Here is the call graph for this function:

bool MarshalStream::VisitNone ( const PyNone rep)
protectedvirtual

Adds a None object to the stream.

Reimplemented from PyVisitor.

Definition at line 145 of file EVEMarshal.cpp.

References Op_PyNone.

146 {
147  Put<uint8>( Op_PyNone );
148  return true;
149 }
bool MarshalStream::VisitObject ( const PyObject rep)
protectedvirtual

Adds an object to the stream.

Reimplemented from PyVisitor.

Definition at line 269 of file EVEMarshal.cpp.

References Op_PyObject, and PyVisitor::VisitObject().

270 {
271  Put<uint8>( Op_PyObject );
272  return PyVisitor::VisitObject( rep );
273 }
virtual bool VisitObject(const PyObject *rep)
Object type visitor.
Definition: PyVisitor.cpp:69

Here is the call graph for this function:

bool MarshalStream::VisitObjectEx ( const PyObjectEx rep)
protectedvirtual

Adds a New object to the stream.

Reimplemented from PyVisitor.

Definition at line 275 of file EVEMarshal.cpp.

References PyList::begin(), PyDict::begin(), PyObjectEx::dict(), PyList::end(), PyDict::end(), PyObjectEx::header(), PyObjectEx::isType2(), PyObjectEx::list(), Op_PackedTerminator, Op_PyObjectEx1, Op_PyObjectEx2, and PyRep::visit().

276 {
277  if (rep->isType2())
278  Put<uint8>( Op_PyObjectEx2 );
279  else
280  Put<uint8>( Op_PyObjectEx1 );
281 
282  if ( !rep->header()->visit( *this ) )
283  return false;
284 
285  PyObjectEx::const_list_iterator lcur = rep->list().begin(), lend = rep->list().end();
286  for (; lcur != lend; ++lcur ) {
287  if ( !(*lcur )->visit( *this ) )
288  return false;
289  }
290 
291  Put<uint8>( Op_PackedTerminator );
292 
293  PyObjectEx::const_dict_iterator dcur = rep->dict().begin(), dend = rep->dict().end();
294  for (; dcur != dend; ++dcur ) {
295  if ( !dcur->first->visit( *this ) )
296  return false;
297  if ( !dcur->second->visit( *this ) )
298  return false;
299  }
300 
301  Put<uint8>( Op_PackedTerminator );
302 
303  return true;
304 }
list_type & list()
Definition: PyRep.h:889
list_type::const_iterator const_list_iterator
Definition: PyRep.h:866
const_iterator begin() const
Definition: PyRep.h:660
dict_type::const_iterator const_dict_iterator
Definition: PyRep.h:870
dict_type & dict()
Definition: PyRep.h:892
const_iterator begin() const
Definition: PyRep.h:766
const_iterator end() const
Definition: PyRep.h:661
bool isType2() const
Definition: PyRep.h:887
PyRep * header() const
Definition: PyRep.h:886
const_iterator end() const
Definition: PyRep.h:767
virtual bool visit(PyVisitor &v) const =0
Visits object.

Here is the call graph for this function:

bool MarshalStream::VisitPackedRow ( const PyPackedRow pyPackedRow)
protectedvirtual

Adds a packed row to the stream.

Reimplemented from PyVisitor.

Definition at line 306 of file EVEMarshal.cpp.

References Buffer::Append(), Buffer::AppendSeq(), PyRep::AsBool(), PyRep::AsFloat(), PyRep::AsInt(), PyRep::AsLong(), Buffer::begin(), DBTYPE_BOOL, DBTYPE_CY, DBTYPE_FILETIME, DBTYPE_GetSizeBits(), DBTYPE_I1, DBTYPE_I2, DBTYPE_I4, DBTYPE_I8, DBTYPE_R4, DBTYPE_R8, DBTYPE_UI1, DBTYPE_UI2, DBTYPE_UI4, DBTYPE_UI8, Buffer::end(), PyPackedRow::GetField(), PyPackedRow::header(), PyRep::IsNone(), Op_PyPackedRow, Buffer::Reserve(), SaveRLE(), EvE::traceStack(), PyInt::value(), PyLong::value(), PyFloat::value(), PyBool::value(), PyRep::visit(), and PyObjectEx::visit().

307 {
308  Put<uint8>( Op_PyPackedRow );
309 
310  DBRowDescriptor* header(pyPackedRow->header());
311  header->visit( *this );
312 
313  // create the sizemap and sort it by bitsize, the value of the map indicates the index of the column
314  // this can be used to identify things easily
315  std::multimap< uint8, uint32, std::greater< uint8 > > sizeMap;
316  std::map<uint8,uint8> booleanColumns;
317 
318  uint32 columnCount = header->ColumnCount();
319  size_t byteDataBitLength = 0;
320  size_t booleansBitLength = 0;
321  size_t nullsBitLength = 0;
322 
323  // go through all the columns to gather the required information
324  for (uint32_t i = 0; i < columnCount; i ++)
325  {
326  DBTYPE columnType = header->GetColumnType (i);
327  uint8_t size = DBTYPE_GetSizeBits (columnType);
328 
329  // count booleans
330  if (columnType == DBTYPE_BOOL)
331  {
332  // register the boolean in the list and increase the length
333  booleanColumns.insert (std::make_pair (i, booleansBitLength));
334  booleansBitLength ++;
335  }
336 
337  // also count all columns as possible nulls
338  nullsBitLength ++;
339 
340  // increase the bytedata length only if a column is longer than 7 bits
341  // this is used as an indicator of what is written in the first second, or third part
342  if (size >= 8)
343  byteDataBitLength += size;
344 
345  // add the column to the list
346  sizeMap.insert (std::make_pair (size, i));
347  }
348 
349  // reserve the space for the buffers
350  Buffer rowData;
351  rowData.Reserve<uint8> ((byteDataBitLength >> 3) + ((booleansBitLength + nullsBitLength) >> 3) + 1);
352  // to ease working with the bit data, reserve a fixed-size buffer
353  // and fill it with 0s
354  Buffer bitData(((booleansBitLength + nullsBitLength) >> 3) + 1, 0);
355 
356  std::multimap< uint8, uint32, std::greater< uint8 > >::iterator cur, end;
357  cur = sizeMap.begin();
358  // limit the search to booleans, the rest of the values are encoded differently
359  end = sizeMap.lower_bound( 1 );
360  PyRep* value(nullptr);
361  for (; cur != end; ++cur)
362  {
363  value = pyPackedRow->GetField(cur->second);
364 
365  // handle the column being none
366  if (value->IsNone() == true)
367  {
368  // get the bit this column should be written at
369  unsigned long nullBit = cur->second + booleansBitLength;
370  unsigned long nullByte = nullBit >> 3;
371  // setup the iterator to the proper byte
372  Buffer::iterator<uint8> bitIterator = bitData.begin<uint8>() + nullByte;
373  // update the proper bit
374  *bitIterator |= (1 << (nullBit & 0x7));
375  }
376 
377  // ensure that the proper value is written
378  // the values will be ignored if a none flag is set, but they must be present
379  switch (header->GetColumnType (cur->second))
380  {
381  case DBTYPE_CY:
382  case DBTYPE_I8:
383  case DBTYPE_UI8:
384  case DBTYPE_FILETIME:
385  rowData.Append<int64>(value->IsNone() ? 0 : value->AsLong()->value() );
386  break;
387  case DBTYPE_I4:
388  rowData.Append<int32>(value->IsNone() ? 0 : value->AsInt()->value() );
389  break;
390  case DBTYPE_UI4:
391  rowData.Append<uint32>(value->IsNone() ? 0 : value->AsInt()->value() );
392  break;
393  case DBTYPE_I2:
394  rowData.Append<int16>(value->IsNone() ? 0 : value->AsInt()->value() );
395  break;
396  case DBTYPE_UI2:
397  rowData.Append<uint16>(value->IsNone() ? 0 : value->AsInt()->value() );
398  break;
399  case DBTYPE_I1:
400  rowData.Append<int8>(value->IsNone() ? 0 : value->AsInt()->value() );
401  break;
402  case DBTYPE_UI1:
403  rowData.Append<uint8>(value->IsNone() ? 0 : value->AsInt()->value() );
404  break;
405  case DBTYPE_R8:
406  rowData.Append<double>(value->IsNone() ? 0.0 : value->AsFloat()->value() );
407  break;
408  case DBTYPE_R4:
409  rowData.Append<float>(static_cast<float>(value->IsNone() ? 0.0f : value->AsFloat()->value()));
410  break;
411  // FIXME nothing should hit here ever but better implement some error-handling just in case
412  default:
413  assert( false );
414  EvE::traceStack();
415  break;
416  }
417  }
418 
419  cur = sizeMap.lower_bound( 1 );
420  end = sizeMap.lower_bound( 0 );
421  PyBool* b(nullptr);
422  for (; cur != end; ++cur)
423  {
424  b = pyPackedRow->GetField(cur->second)->AsBool();
425 
426  // false values do not need anything to be done
427  if (b->value() == false)
428  continue;
429 
430  // get the bit this boolean should be written at
431  unsigned long boolBit = booleanColumns.find (cur->second)->second;
432  unsigned long boolByte = boolBit >> 3;
433  // setup the iterator to the proper byte
434  Buffer::iterator<uint8> bitIterator = bitData.begin<uint8>() + boolByte;
435  // update the proper bit
436  *bitIterator |= (1 << (boolBit & 0x7));
437  }
438 
439  // concatenate the bit data to the rowData
440  rowData.AppendSeq(bitData.begin<uint8>(), bitData.end<uint8>());
441 
442  // run the data through the zero compression algorithm
443  if (!SaveRLE(rowData))
444  return false;
445 
446  // finally append items that are not packed like strings or byte buffers
447  cur = sizeMap.lower_bound( 0 );
448  end = sizeMap.end();
449  for (; cur != end; ++cur) {
450  value = pyPackedRow->GetField(cur->second );
451  if (!value->visit(*this))
452  return false;
453  }
454 
455  return true;
456 }
void Append(const T &value)
Appends a single value to buffer.
Definition: Buffer.h:437
Base Python wire object.
Definition: PyRep.h:66
unsigned __int8 uint8
Definition: eve-compat.h:46
bool visit(PyVisitor &v) const
Visits object.
Definition: PyRep.cpp:804
PyBool * AsBool()
Definition: PyRep.h:128
DBRowDescriptor * header() const
Definition: PyRep.h:983
signed __int8 int8
Definition: eve-compat.h:45
signed __int32 int32
Definition: eve-compat.h:49
Python boolean.
Definition: PyRep.h:323
PyRep * GetField(size_t index) const
Definition: PyRep.h:990
Generic class for buffers.
Definition: Buffer.h:40
void AppendSeq(Iter first, Iter last)
Appends a sequence of elements to buffer.
Definition: Buffer.h:455
Python object "blue.DBRowDescriptor".
Definition: PyDatabase.h:41
DBTYPE
Definition: dbtype.h:67
void Reserve(size_type requiredCount)
Reserves (pre-allocates) memory for buffer.
Definition: Buffer.h:626
unsigned __int32 uint32
Definition: eve-compat.h:50
uint8 DBTYPE_GetSizeBits(DBTYPE type)
Definition: dbtype.cpp:30
signed __int64 int64
Definition: eve-compat.h:51
bool SaveRLE(const Buffer &in)
Definition: EVEMarshal.cpp:524
signed __int16 int16
Definition: eve-compat.h:47
void traceStack(void)
Definition: misc.cpp:169
unsigned __int16 uint16
Definition: eve-compat.h:48
Buffer's iterator.
Definition: Buffer.h:232

Here is the call graph for this function:

bool MarshalStream::VisitReal ( const PyFloat rep)
protectedvirtual

Adds a double to the stream.

Reimplemented from PyVisitor.

Definition at line 133 of file EVEMarshal.cpp.

References Op_PyReal, Op_PyZeroReal, and PyFloat::value().

134 {
135  if ( rep->value() == 0.0 ) {
136  Put<uint8>( Op_PyZeroReal );
137  } else {
138  Put<uint8>( Op_PyReal );
139  Put<double>( rep->value() );
140  }
141 
142  return true;
143 }
double value() const
Definition: PyRep.h:309

Here is the call graph for this function:

bool MarshalStream::VisitString ( const PyString rep)
protectedvirtual

add a string object to the data stream

Reimplemented from PyVisitor.

Definition at line 163 of file EVEMarshal.cpp.

References PyString::content(), Op_PyCharString, Op_PyEmptyString, Op_PyLongString, Op_PyStringTableItem, Put(), PutSizeEx(), sMarshalStringTable, and STRING_TABLE_ERROR.

164 {
165  size_t len(rep->content().size());
166 
167  if ( len == 0 ) {
168  Put<uint8>( Op_PyEmptyString );
169  } else if ( len == 1 ) {
170  Put<uint8>( Op_PyCharString );
171  Put<uint8>( rep->content()[0] );
172  } else {
173  //string is long enough for a string table entry, check it.
174  const uint8 index = sMarshalStringTable.LookupIndex( rep->content() );
175  if ( index > STRING_TABLE_ERROR ) {
176  Put<uint8>( Op_PyStringTableItem );
177  Put<uint8>( index );
178  } else {
179  // NOTE: they seem to have stopped using Op_PyShortString
180  Put<uint8>( Op_PyLongString );
181  PutSizeEx( (uint32)len );
182  Put( rep->content().begin(), rep->content().end() );
183  }
184  }
185 
186  return true;
187 }
unsigned __int8 uint8
Definition: eve-compat.h:46
#define sMarshalStringTable
void Put(const T &value)
Definition: EVEMarshal.h:75
unsigned __int32 uint32
Definition: eve-compat.h:50
#define STRING_TABLE_ERROR
const std::string & content() const
Get the PyString content.
Definition: PyRep.h:458
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitSubStream ( const PySubStream rep)
protectedvirtual

Adds a sub stream to the stream.

Reimplemented from PyVisitor.

Definition at line 464 of file EVEMarshal.cpp.

References Buffer::begin(), PyBuffer::content(), PySubStream::data(), PySubStream::decoded(), PySubStream::EncodeData(), Buffer::end(), Op_PySubStream, Put(), PutSizeEx(), and Buffer::size().

465 {
466  Put<uint8>(Op_PySubStream);
467  if (rep->data() == nullptr) {
468  if (rep->decoded() == nullptr) {
469  Put<uint8>(0);
470  return false;
471  }
472 
473  //unmarshaled stream
474  //we have to marshal the substream.
475  rep->EncodeData();
476  if (rep->data() == nullptr) {
477  Put<uint8>(0);
478  return false;
479  }
480  }
481 
482  //we have the marshaled data, use it.
483  const Buffer& data = rep->data()->content();
484 
485  PutSizeEx( (uint32)data.size() );
486  Put( data.begin<uint8>(), data.end<uint8>() );
487 
488  return true;
489 }
unsigned __int8 uint8
Definition: eve-compat.h:46
const Buffer & content() const
Get the const PyBuffer content.
Definition: PyRep.h:407
PyBuffer * data() const
Definition: PyRep.h:1047
PyRep * decoded() const
Definition: PyRep.h:1048
iterator< T > begin()
Definition: Buffer.h:381
Generic class for buffers.
Definition: Buffer.h:40
void Put(const T &value)
Definition: EVEMarshal.h:75
unsigned __int32 uint32
Definition: eve-compat.h:50
size_type size() const
Definition: Buffer.h:610
void EncodeData() const
Definition: PyRep.cpp:1109
iterator< T > end()
Definition: Buffer.h:387
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitSubStruct ( const PySubStruct rep)
protectedvirtual

Adds a sub structure to the stream.

Reimplemented from PyVisitor.

Definition at line 458 of file EVEMarshal.cpp.

References Op_PySubStruct, and PyVisitor::VisitSubStruct().

459 {
460  Put<uint8>(Op_PySubStruct);
461  return PyVisitor::VisitSubStruct( rep );
462 }
virtual bool VisitSubStruct(const PySubStruct *rep)
wrapper types Visitor
Definition: PyVisitor.cpp:112

Here is the call graph for this function:

bool MarshalStream::VisitToken ( const PyToken rep)
protectedvirtual

add a token object to the data stream

Reimplemented from PyVisitor.

Definition at line 207 of file EVEMarshal.cpp.

References PyToken::content(), Op_PyToken, Put(), and PutSizeEx().

208 {
209  Put<uint8>( Op_PyToken );
210 
211  const std::string& str = rep->content();
212 
213  PutSizeEx( (uint32)str.size() );
214  Put( str.begin(), str.end() );
215 
216  return true;
217 }
void Put(const T &value)
Definition: EVEMarshal.h:75
unsigned __int32 uint32
Definition: eve-compat.h:50
const std::string & content() const
Obtain token.
Definition: PyRep.h:555
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitTuple ( const PyTuple rep)
protectedvirtual

Add a tuple object to the stream

Reimplemented from PyVisitor.

Definition at line 219 of file EVEMarshal.cpp.

References Op_PyEmptyTuple, Op_PyOneTuple, Op_PyTuple, Op_PyTwoTuple, PutSizeEx(), PyTuple::size(), and PyVisitor::VisitTuple().

220 {
221  uint32 size(rep->size());
222  if ( size == 0 ) {
223  Put<uint8>( Op_PyEmptyTuple );
224  } else if ( size == 1 ) {
225  Put<uint8>( Op_PyOneTuple );
226  } else if ( size == 2 ) {
227  Put<uint8>( Op_PyTwoTuple );
228  } else {
229  Put<uint8>( Op_PyTuple );
230  PutSizeEx( size );
231  }
232 
233  return PyVisitor::VisitTuple( rep );
234 }
virtual bool VisitTuple(const PyTuple *rep)
the nested types Visitor
Definition: PyVisitor.cpp:36
size_t size() const
Definition: PyRep.h:591
unsigned __int32 uint32
Definition: eve-compat.h:50
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

bool MarshalStream::VisitWString ( const PyWString rep)
protectedvirtual

add a wide string object to the data stream

Reimplemented from PyVisitor.

Definition at line 189 of file EVEMarshal.cpp.

References PyWString::content(), Op_PyEmptyWString, Op_PyWStringUTF8, Put(), and PutSizeEx().

190 {
191  size_t len(rep->content().size());
192 
193  if ( len == 0 ) {
194  Put<uint8>( Op_PyEmptyWString );
195  } else {
196  // We don't have to consider any conversions because
197  // UTF-8 is more space-efficient than UCS-2.
198 
199  Put<uint8>( Op_PyWStringUTF8 );
200  PutSizeEx( (uint32)len );
201  Put( rep->content().begin(), rep->content().end() );
202  }
203 
204  return true;
205 }
const std::string & content() const
Get the PyWString content.
Definition: PyRep.h:499
void Put(const T &value)
Definition: EVEMarshal.h:75
unsigned __int32 uint32
Definition: eve-compat.h:50
void PutSizeEx(uint32 size)
Definition: EVEMarshal.h:81

Here is the call graph for this function:

Member Data Documentation

Buffer* MarshalStream::mBuffer
private

Definition at line 146 of file EVEMarshal.h.

Referenced by Put(), and Save().


The documentation for this class was generated from the following files: