Skip to content

Added cached properties implementation. #292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 81 commits into from
Jun 20, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
3842149
Added cached properties implementation.
jordanbriere Nov 28, 2019
d90de7c
Fixed cached generators being exhausted after the first iteration.
jordanbriere Nov 29, 2019
7aefd2a
Fixed CachedGenerator not caching all the generated values if the fir…
jordanbriere Nov 29, 2019
fe034ab
Dynamic entity functions are now cached.
jordanbriere Nov 29, 2019
af24025
Added Entity.instances cached property.
jordanbriere Nov 30, 2019
e06bde9
Entity.inputs and Entity.outputs instances are now cached.
jordanbriere Nov 30, 2019
70717da
Improved performance of Entity.<get/set>_property_<type> methods.
jordanbriere Dec 1, 2019
a4642cc
Fixed some memory leaks.
jordanbriere Dec 1, 2019
afde4a1
Fixed CachedProperty's decorators overriding the main descriptor name.
jordanbriere Dec 2, 2019
388e71e
EntityMemFuncWrapper is no longer caching itself; the Entity class al…
jordanbriere Dec 3, 2019
9c7e4a2
Fixed Player.is_bot and Player.is_hltv now being properties rather th…
jordanbriere Dec 3, 2019
9066933
Improved the caching of dynamic function wrappers.
jordanbriere Dec 3, 2019
b187391
Fixed a circular reference.
jordanbriere Dec 4, 2019
682ee74
Fixed cached properties declared on the c++ side not being managed by…
jordanbriere Dec 4, 2019
0e29616
Removed CachedProperty.owner, which was causing another circular refe…
jordanbriere Dec 4, 2019
258c105
Fixed back reference issues.
jordanbriere Dec 4, 2019
fc618e1
Fixed KeyError when invalidating the cache for field that were not pr…
jordanbriere Dec 4, 2019
de055b8
Fixed EntityDictionary potentially caching soon-to-be removed entities.
jordanbriere Dec 4, 2019
b664ef0
Added a warning to CachedProperty's documentation regarding circular …
jordanbriere Dec 4, 2019
be36626
Fixed dynamic function wrappers from no longer being documented.
jordanbriere Dec 4, 2019
7747961
Fixed Entity.__setattr__ not properly iterating over the entity's ser…
jordanbriere Dec 5, 2019
88bec54
Added "unbound" parameter to CachedProperty as a workaround for circu…
jordanbriere Dec 6, 2019
45a4060
Added back CachedProperty.owner as a weak reference.
jordanbriere Dec 6, 2019
8502e1b
Fixed Player.is_bot to be cached correctly. (#294)
CookStar Dec 11, 2019
debc63b
Fixed a crash caused by Function's convention being freed twice when …
jordanbriere Dec 14, 2019
02ea4e1
Fixed the invalidation of the internal entity cache before all entity…
jordanbriere Dec 25, 2019
a74aa97
Merge branch 'master' into cached_property
jordanbriere Jan 3, 2020
44f89f6
Added caching boolean argument for Player.from_userid.
satoon101 Jan 4, 2020
f90078c
Disabled instance caching for Entity's subclasses that do not explici…
jordanbriere Jan 9, 2020
545ef0d
Improved performance of memory tools (approximately 2.5 times faster).
jordanbriere Jan 9, 2020
31ea73a
Merge branch 'master' into bugfix_player_from_userid
jordanbriere Jan 9, 2020
f87788a
Merge branch 'bugfix_player_from_userid' into cached_property
jordanbriere Jan 9, 2020
ecab5bb
Merge branch 'master' into cached_property
jordanbriere Jan 9, 2020
7c5f43a
Merge branch 'master' into cached_property
jordanbriere Mar 11, 2020
6e8d282
Merge branch 'master' into cached_property
jordanbriere Mar 18, 2020
d8043ba
Merge branch 'master' into cached_property
jordanbriere Apr 9, 2020
5130d5c
Fixed exceptions potentially being silenced when caching generators.
jordanbriere Apr 9, 2020
abba9e7
Added missing args and kwargs parameters to CachedProperty.wrap_descr…
jordanbriere Apr 9, 2020
37c9337
Merge remote-tracking branch 'remotes/origin/master' into cached_prop…
jordanbriere Apr 10, 2020
6a6dd96
Merge remote-tracking branch 'remotes/origin/master' into cached_prop…
jordanbriere Apr 15, 2020
6f55b09
Merge branch 'master' into cached_property
jordanbriere Apr 24, 2020
274faf4
Merge branch 'master' into cached_property
jordanbriere Apr 28, 2020
3785faa
Merge branch 'master' into cached_property
jordanbriere May 1, 2020
b6ed444
Merge branch 'master' into cached_property
jordanbriere May 2, 2020
03609da
Merge branch 'master' into cached_property
jordanbriere May 2, 2020
2c309b5
Moved an extraction to avoid doing it when not necessary.
jordanbriere May 2, 2020
e9c5a82
Optimized Entity.<get/set>_property_edict methods.
jordanbriere May 2, 2020
6e84aac
Merge remote-tracking branch 'remotes/origin/master' into cached_prop…
jordanbriere May 4, 2020
fa1926b
Fixed entity delays/repeats not being cancelled if they were register…
jordanbriere May 4, 2020
3cedd95
Player.language property is now only cached for games it was already …
jordanbriere May 4, 2020
b8afdb6
Optimized non-cached get_client_language by using the internal Player…
jordanbriere May 4, 2020
24380ff
Optimized various python calls from c++.
jordanbriere May 4, 2020
ea77ce4
Optimized pointer extraction.
jordanbriere May 4, 2020
0c4248e
Fixed memory leaks caused by Sound/StreamSound instances never unload…
jordanbriere May 4, 2020
f3e8ce5
Fixed memory leaks into engine_sound.emit_sound caused by the sounds …
jordanbriere May 4, 2020
fe1422f
Merge branch 'master' into cached_property
jordanbriere May 4, 2020
95cf384
Merge branch 'master' into cached_property
jordanbriere May 5, 2020
a179e8c
Improved performance of Entity.is_in_solid's default behaviour (appro…
jordanbriere May 5, 2020
a87f121
Moved Entity.<get/set>_property_<type> methods to BaseEntity (approx.…
jordanbriere May 5, 2020
ff4ec27
Removed Entity.<get/set>_property_<type> methods.
jordanbriere May 5, 2020
8d1ed6e
Merge branch 'master' into cached_property
jordanbriere May 6, 2020
4269a14
Improved performance of TraceFilterSimple.should_hit_entity (approx. …
jordanbriere May 6, 2020
6fcf7b1
Oops, fixed a condition.
jordanbriere May 6, 2020
c613ac8
Added example to CachedProperty's docstring.
jordanbriere May 7, 2020
705a088
Improved performance of Entity.create, find and find_or_create by mov…
jordanbriere May 7, 2020
6fe0038
Fixed non-networked entities creation/lookup.
jordanbriere May 7, 2020
ec6ac90
Added an overload to memory.make_object to avoid redundant extraction.
jordanbriere May 7, 2020
bb86856
Added BaseEntity.is_marked_for_deletion (approx. 13 times faster than…
jordanbriere May 7, 2020
83e23cb
Merge branch 'master' into cached_property
jordanbriere May 8, 2020
1b67129
Fixed BaseEntity.is_marked_for_deletion from checking the wrong flag.
jordanbriere May 8, 2020
3e7b711
Moved InputFunction to c++ (~21x faster).
jordanbriere May 10, 2020
067d6ff
Added sanity check to InputFunction's constructor.
jordanbriere May 10, 2020
3809139
Removed no longer used imports into entities._base.
jordanbriere May 10, 2020
350825f
Merge branch 'master' into cached_property
jordanbriere May 14, 2020
1265508
Added cached_result decorator.
jordanbriere May 16, 2020
4cb773c
Merge branch 'master' into cached_property
jordanbriere May 16, 2020
2ce7227
Removed a redundant layer to get Entity.index.
jordanbriere May 16, 2020
79e62a7
Added CachedProperty.<get/set>_cached_value methods.
jordanbriere May 17, 2020
b9e8826
Fixed CachedProperty.<get/set>_cached_value methods from possibly cac…
jordanbriere May 19, 2020
f730a62
Improved some Entity's methods by avoiding repeated attribute retriev…
jordanbriere May 19, 2020
fe1f31c
Merge branch 'master' into cached_property
jordanbriere May 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixed some memory leaks.
  • Loading branch information
jordanbriere committed Dec 1, 2019
commit a4642ccb0cf412de723c0998ce3092a7ffd83b41
10 changes: 8 additions & 2 deletions addons/source-python/packages/source-python/entities/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -339,8 +339,11 @@ def server_classes(self):
@cached_property
def properties(self):
"""Iterate over all descriptors available for the entity."""
properties = {}
for server_class in self.server_classes:
yield from server_class.properties
for prop, data in server_class.properties.items():
properties[prop] = data
return properties

@cached_property
def inputs(self):
Expand Down Expand Up @@ -374,8 +377,11 @@ def keyvalues(self):
An entity might also have hardcoded keyvalues that can't be listed
with this property.
"""
keyvalues = {}
for server_class in self.server_classes:
yield from server_class.keyvalues
for keyvalue, data in server_class.keyvalues.items():
keyvalues[keyvalue] = data
return keyvalues

def get_model(self):
"""Return the entity's model.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,11 @@ class InputFunction(Function):

def __init__(self, name, argument_type, function, this):
"""Instantiate the function instance and store the base attributes."""
super().__init__(function)
self._function = function
super().__init__(
function.address, function.convention, function.arguments,
function.return_type
)

self._name = name
self._argument_type = argument_type
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,10 @@ class MemberFunction(Function):

def __init__(self, manager, return_type, func, this):
"""Initialize the instance."""
super().__init__(func)
self._function = func
super().__init__(
func.address, func.convention, func.arguments, func.return_type
)

# This should always hold a TypeManager instance
self._manager = manager
Expand Down
28 changes: 28 additions & 0 deletions src/core/modules/memory/memory_function.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,9 @@ CFunction::CFunction(unsigned long ulAddr, object oCallingConvention, object oAr
// If this line succeeds the user wants to create a function with the built-in calling conventions
m_eCallingConvention = extract<Convention_t>(oCallingConvention);
m_pCallingConvention = MakeDynamicHooksConvention(m_eCallingConvention, ObjectToDataTypeVector(m_tArgs), m_eReturnType);

// We allocated the calling convention, we are responsible to cleanup.
m_bAllocatedCallingConvention = true;
}
catch( ... )
{
Expand All @@ -154,8 +157,12 @@ CFunction::CFunction(unsigned long ulAddr, object oCallingConvention, object oAr
// FIXME:
// This is required to fix a crash, but it will also cause a memory leak,
// because no calling convention object that is created via this method will ever be deleted.
// TODO: Pretty sure this was required due to the missing held type definition. It was added, but wasn't tested yet.
Py_INCREF(_oCallingConvention.ptr());
m_pCallingConvention = extract<ICallingConvention*>(_oCallingConvention);

// We didn't allocate the calling convention, someone else is responsible for it.
m_bAllocatedCallingConvention = false;
}

// Step 4: Get the DynCall calling convention
Expand All @@ -170,11 +177,32 @@ CFunction::CFunction(unsigned long ulAddr, Convention_t eCallingConvention,
m_eCallingConvention = eCallingConvention;
m_iCallingConvention = iCallingConvention;
m_pCallingConvention = pCallingConvention;

// We didn't allocate the calling convention, someone else is responsible for it.
m_bAllocatedCallingConvention = false;

m_tArgs = tArgs;
m_eReturnType = eReturnType;
m_oConverter = oConverter;
}

CFunction::~CFunction()
{
// If we didn't allocate the calling convention, then it is not our responsibility.
if (!m_bAllocatedCallingConvention)
return;

CHook* pHook = GetHookManager()->FindHook((void *) m_ulAddr);

// DynamicHooks will take care of it for us from there.
if (pHook && pHook->m_pCallingConvention == m_pCallingConvention)
return;

// Cleanup.
delete m_pCallingConvention;
m_pCallingConvention = NULL;
}

bool CFunction::IsCallable()
{
return (m_eCallingConvention != CONV_CUSTOM) && (m_iCallingConvention != -1);
Expand Down
5 changes: 4 additions & 1 deletion src/core/modules/memory/memory_function.h
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,16 @@ enum Convention_t
// ============================================================================
// >> CFunction
// ============================================================================
class CFunction: public CPointer
class CFunction: public CPointer, private boost::noncopyable
{
public:
CFunction(unsigned long ulAddr, object oCallingConvention, object oArgs, object oReturnType);
CFunction(unsigned long ulAddr, Convention_t eCallingConvention, int iCallingConvention,
ICallingConvention* pCallingConvention, boost::python::tuple tArgs,
DataType_t eReturnType, object oConverter);

~CFunction();

bool IsCallable();
bool IsHookable();

Expand Down Expand Up @@ -99,6 +101,7 @@ class CFunction: public CPointer

// DynamicHooks calling convention (built-in and custom)
ICallingConvention* m_pCallingConvention;
bool m_bAllocatedCallingConvention;
};


Expand Down
13 changes: 4 additions & 9 deletions src/core/modules/memory/memory_pointer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -109,14 +109,6 @@ void CPointer::SetStringArray(char* szText, int iOffset /* = 0 */)
EXCEPT_SEGV()
}

unsigned long GetPtrHelper(unsigned long addr)
{
TRY_SEGV()
return *(unsigned long *) addr;
EXCEPT_SEGV()
return 0;
}

CPointer* CPointer::GetPtr(int iOffset /* = 0 */)
{
Validate();
Expand Down Expand Up @@ -286,7 +278,10 @@ CFunction* CPointer::MakeFunction(object oCallingConvention, object oArgs, objec

CFunction* CPointer::MakeVirtualFunction(int iIndex, object oCallingConvention, object args, object return_type)
{
return GetVirtualFunc(iIndex)->MakeFunction(oCallingConvention, args, return_type);
CPointer *pPtr = GetVirtualFunc(iIndex);
CFunction *pFunc = pPtr->MakeFunction(oCallingConvention, args, return_type);
delete pPtr;
return pFunc;
}

CFunction* CPointer::MakeVirtualFunction(CFunctionInfo& info)
Expand Down
12 changes: 12 additions & 0 deletions src/core/modules/memory/memory_pointer.h
Original file line number Diff line number Diff line change
Expand Up @@ -190,4 +190,16 @@ inline CPointer* Alloc(int iSize, bool bAutoDealloc = true)
return new CPointer((unsigned long) UTIL_Alloc(iSize), bAutoDealloc);
}


// ============================================================================
// >> GetPtrHelper
// ============================================================================
inline unsigned long GetPtrHelper(unsigned long addr)
{
TRY_SEGV()
return *(unsigned long *) addr;
EXCEPT_SEGV()
return 0;
}

#endif // _MEMORY_POINTER_H
2 changes: 1 addition & 1 deletion src/core/modules/memory/memory_scanner.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,7 @@ CPointer* CBinaryFile::FindPointer(object oIdentifier, int iOffset, unsigned int
ptr->m_ulAddr += iOffset;
while (iLevel > 0)
{
ptr = ptr->GetPtr();
ptr->m_ulAddr = GetPtrHelper(ptr->m_ulAddr);
iLevel = iLevel - 1;
}
}
Expand Down
5 changes: 3 additions & 2 deletions src/core/modules/memory/memory_wrap.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -452,7 +452,8 @@ void export_type_info_iter(scope _memory)
void export_function(scope _memory)
{
class_<CFunction, bases<CPointer>, boost::noncopyable >("Function", init<unsigned long, object, object, object>())
.def(init<CFunction&>())
// Don't allow copies, because they will hold references to our calling convention.
// .def(init<CFunction&>())
.def("__call__",
raw_method(&CFunction::Call),
"Calls the function dynamically."
Expand Down Expand Up @@ -817,7 +818,7 @@ void export_registers(scope _memory)
// ============================================================================
void export_calling_convention(scope _memory)
{
class_<ICallingConventionWrapper, boost::noncopyable>(
class_<ICallingConventionWrapper, ICallingConventionWrapper *, boost::noncopyable>(
"CallingConvention",
"An an abstract class that is used to create custom calling "
"conventions (only available for hooking function and not for"
Expand Down