USOO932,3931B2

(12) United States Patent (10) Patent No.: US 9,323,931 B2

Lukacs et al. (45) Date of Patent: Apr. 26, 2016

(54) COMPLEX SCORING FORMALWARE 7,870,612 B2 1/2011 Liu

DETECTION 7,908,660 B2 3, 2011 Bahl
7,930,751 B2 4/2011 Alagna et al.
8,037,536 B2 10/2011 Treadwell et al.
(71) Applicant: Bitdefender IPR Management Ltd., 8,578,345 B1 * 1 1/2013 Kennedy et al. .............. 717 132
Nicosia (CY) 8,615,805 B1* 12/2013 Obrecht et al. ................. T26/24
(72) Inventors: Sandor Lukacs, Floresti (RO); Raul V. (Continued)
Tosa, Cluj-Napoca (RO); Paul Boca,
Bistrita (RO); Gheorghe Hajimasan, FOREIGN PATENT DOCUMENTS
Lunca Muresului (RO); Andrei V. Lutas, WO 2010O23557 A2 3, 2010
Satu Mare (RO) OTHER PUBLICATIONS
(73) Assignee: siderster Management Ltd., Bitclefender, "BitDefender Active Virus Control: Pro Active Protec
tion Against New and Emerging Threats,” whitepaper, p. 1-10,
(*) Notice: Subject to any disclaimer, the term of this Bitclefender Antivirus Software, Bucharest, Romania; Jan. 2010.
patent is extended or adjusted under 35 (Continued)
U.S.C. 154(b) by 86 days.
Primary Examiner — Izunna Okeke
(21) Appl. No.: 14/046,728 Assistant Examiner — Hee Song
(22) Filed: Oct. 4, 2013 E. Attorney, Agent, or Firm — Law Office of Andrei D
opovici, PC
(65) Prior Publication Data (57) ABSTRACT
US 2015/O1 O1049 A1 Apr. 9, 2015 Described systems and methods allow protecting a computer
(51) Int. Cl. system from malware Such as viruses, Trojans, and spyware.
H04L 29/06 (2006.01) For each of a plurality of executable entities (such as pro
G06F 2/56 (2013.01)
cesses and threads executing on the computer System), a
scoring engine records a plurality of evaluation scores, each
(52) U.S. Cl. score determined according to a distinct evaluation criterion.
CPC .............. G06F2I/566 (2013.01); H04L 63/14 Every time an entity satisfies an evaluation criterion (e.g.,
(2013.01) perf h respective
erforms an action), the pecti Score off the
th ent1ty
ity i1S
(58) Field of Classification Search updated. Updating a score of an entity may trigger score
CPC ..................................................... GO6F 21 F566 updates of entities related to the respective entity, even when
USPC .. . . .. . . . . . .. . . . . . 726/22 25 the related entities are terminated, i.e., no longer active.
See application file for complete search history. Related entities include, among others, a parent of the respec
(56) References Cited
tive entity, and/or an entity injecting code into the respective
entity. The scoring engine determines whether an entity is
U.S. PATENT DOCUMENTS malicious according to the plurality of evaluation scores of
the respective entity.
7,620,992 B2 11/2009 Monastyrsky et al.
7,748,039 B2 6/2010 Alagna et al. 15 Claims, 10 Drawing Sheets

H. H. H. H. H. H.

Virtual
machine User-level
S-IWC
-- entity cValuator
tier &

Aafay42&

- Anti- - Entity
34 Operating Kernel-level .
malwarc mgmt.
System driver cntity cvaluator module

Hypervisor Memory introspection engine

32
30 40
 US 9,323.931 B2
Page 2

(56) References Cited OTHER PUBLICATIONS

U.S. PATENT DOCUMENTS European Patent Office, International Search Report and Written
2004/0230835 A1 11/2004 Goldfeder et al. Opinion of the International Searching Authority, Mailed Feb. 19.
2010, O293615 A1 11, 2010 Ye
2012/031 1708 Al 12/2012 Agarwal et al. 2015 for PCT International Application No. PCT/RO2014/000027,
2012/0324575 A1 12, 2012 Choi et al. Filed Sep. 25, 2014.
2013,0145463 A1* 6, 2013 Ghosh et al. T26/22
2015/0074806 A1* 3/2015 Roundy et al. .................. T26/23 * cited by examiner
U.S. Patent Apr. 26, 2016 Sheet 1 of 10 US 9,323.931 B2

12 24 14

16
Inpu Dutput devices 18
20 Network

1()

FIG. 1

44
42a 42b 42c

Security
App.

10 Host system hardware

FIG. 2-A
U.S. Patent Apr. 26, 2016 Sheet 2 of 10 US 9,323.931 B2

FIG. 2-B

Virtual
machine w
Security
application
Car Ave/

Aerve/Aza/ 5
Ob
Operating malware
Kernel-level fight.
Scoring
Svstem entity evaluator g engine
v driver modulc

30 40
FIG. 3
U.S. Patent Apr. 26, 2016 Sheet 3 of 10 US 9,323.931 B2

Listen for cntity

life cycle events

Identify
triggering entity
YES
256
Add new entity
to collection

Spawned Add child entity

child entity? to COllection

Code Register code injection

injection? relation

Identify descendants

All
descendants
terminated?
YES
27A
Remove entity
from collection

FIG. 4
U.S. Patent Apr. 26, 2016 Sheet 4 of 10 US 9,323.931 B2

----------------------------------------
Virtual machine
50a | User-level Evaluation 52a.
entity evaluator indicator 38

5 Ob Kernel-level Evaluation 52b Scoring

entity evaluator indicator engine

5Oc System call Evaluation

evaluator indicator 5 2C

IEvaluation 32
indicator 52d
40
Memory introspection engine

FIG. 5
U.S. Patent Apr. 26, 2016 Sheet 5 of 10 US 9,323.931 B2

72b 7

70a Process 2

70
User-level entity evaluator

kernel32.dll ( Createlile(), Writelile(), OpenProcess(). )

ntdl.dll ( ZwCreateFile(), ZwWriteFile(), Zw OpenProcess(). )

Aewe/Aye/ MSR

Sytc. Far-Sa+ y ti SVStem call

KiSystemService() KilastCall Entry.() yS I C2 50c
With SSDT list (ValuatOf

Kerinel-level NT Kerinel:
NT Kernel: entity ObjectManager,
FltMgr.sys evaluator ProcessManager,
Configuration Manager

Filc system driver

(Virtualized) hardware:
memory, Storage, network adapters ...
FIG. 6
U.S. Patent Apr. 26, 2016 Sheet 6 of 10 US 9,323.931 B2

Listen for trigger event

304
Trigger event detected?

306 Identify entity /t

Determine eval. indicator of entity /t

310 Transmit eval. indicator

of entity /t to scoring engine

FIG. 7
U.S. Patent Apr. 26, 2016 Sheet 7 of 10 US 9,323.931 B2

74C
74a
Entity Scoring
object (ESO) 1

EID. EID 0.
EIDO ED. . .
FIG. 8
U.S. Patent Apr. 26, 2016 Sheet 8 of 10 US 9,323.931 B2

s 82a Y sh- 82C 82d Y s

SCOre Initialization Propagation New instance Exception Flag-induced
values weights weights weights weights weights

Eval.
criteria
U.S. Patent Apr. 26, 2016 Sheet 9 of 10 US 9,323.931 B2

Receive entity eval. indicator

Identify evaluated entity At

according to entity
evaluation indicator

x Takic anti-malware action

Update scores of entity X can ware Xr
against entity

Identify entity A * related to \,

Updatc flags of cntity X wherein scores of X* need to
bc updatcdbcCause of update of X

Compute aggregate score

of entity X

Score
of X exceeds
threshold P

-------------------------------------------- -
U.S. Patent Apr. 26, 2016 Sheet 10 of 10 US 9,323.931 B2

11()

I lost system Security server

Host system
FIG 11

Security report
(> 11()
82

Host system Security server

Security settings

FIG. 12
 US 9,323,931 B2
1. 2
COMPLEX SCORING FORMALWARE ties of a first entity of the collection; determining whether the
DETECTION first entity is terminated; in response, when the first entity is
terminated, determining whether all members of the set of
BACKGROUND descendant entities are terminated; and in response, when all
members of the set of descendant entities are terminated,
The invention relates to systems and methods for protect removing the selected entity from the collection. The instruc
ing computer systems from malware. tions further configure the at least one processor to record a
Malicious Software, also known as malware, affects a great first score determined for the first entity and a second score
number of computer systems worldwide. In its many forms determined for a second entity of the collection, the first and
Such as computer viruses, worms, rootkits, and spyware, mal- 10 second scores determined according to an evaluation crite
ware presents a serious risk to millions of computer users, rion. The instructions further configure the at least one pro
making them vulnerable to loss of data and sensitive infor cessor, in response to recording the first and second scores, to
mation, identity theft, and loss of productivity, among others. evaluate the first entity according to the evaluation criterion.
Security software may be used to detect malware infecting
a user's computer system, and additionally to remove or stop 15 The instructions further configure the at least one processor,
in response to evaluating the first entity, when the first entity
the execution of such malware. Several malware-detection
techniques are known in the art. Some rely on matching a satisfies the evaluation criterion, to update the second score,
fragment of code of the malware agent to a library of mal and in response to updating the second score, to determine
ware-indicative signatures. Other conventional methods whether the second entity is malicious according to the
detect a set of malware-indicative behaviors of the malware 20 updated second score.
agent. According to another aspect, a host system comprises at
To evade detection and/or undermine the operation of secu least one processor configured to execute an entity evaluator
rity Software, Some malware agents employ obfuscation tech and a scoring engine. The entity evaluator is configured to:
niques, such as encrypting their code, or using slightly dif evaluate the first software entity according to an evaluation
ferent code versions on each infected computer system 25 criterion, the first software entity executing on the client
(polymorphism). Other exemplary detection avoidance meth system, and in response, when the first Software entity satis
ods divide malicious activities into several actions, each fies the evaluation criterion, transmit an evaluation indicator
action performed by a separate agent, possibly with a time to the scoring engine. The scoring engine is configured, in
delay. In other examples, malware may try to actively attack response to receiving the evaluation indicator, to update a
and disable the security software, for instance by using privi- 30 score according to the evaluation indicator, wherein the score
lege escalation and/or by overwriting the security Software's is determined for a second Software entity previously execut
code. ing on the host system, the second software entity terminated
To keep up with a rapidly changing set of malware threats, at the time of updating the score. The scoring engine is further
there is a strong interest in developing robust and Scalable configured, in response to updating the second score, to deter
anti-malware solutions. 35 mine whether the second software entity is malicious accord
ing to the updated second score.
SUMMARY According to another aspect, a method comprises employ
ing at least one processor of a host system to determine
According to one aspect, a host system comprises at least whether a first software entity executing on the host system
one processor configured to execute an entity management 40 satisfies an evaluation criterion. The method further com
module, an entity evaluator, and a scoring engine. The entity prises, when the first software entity satisfies the evaluation
management module is configured to manage a collection of criterion, employing the at least one processor to update a
evaluated Software entities, wherein managing the collection score determined for a second software entity previously
comprises: identifying a set of descendant entities of a first executing on the host system, the second software entity
entity of the collection; determining whether the first entity is 45 terminated at the time of updating the score, the score deter
terminated; in response, when the first entity is terminated, mined according to the evaluation criterion. The method fur
determining whether all members of the set of descendant ther comprises, in response to updating the second score,
entities are terminated; and in response, when all members of employing the at least one processor to determine whether the
the set of descendant entities are terminated, removing the second Software entity is malicious according to the updated
first entity from the collection. The entity evaluator is config- 50 second score.
ured to: evaluate the first entity according to an evaluation
criterion; and in response, when the first entity satisfies the BRIEF DESCRIPTION OF THE DRAWINGS
evaluation criterion, transmit an evaluation indicator to the
scoring engine. The scoring engine is configured to: record a The foregoing aspects and advantages of the present inven
first score determined for the first entity and a second score 55 tion will become better understood upon reading the follow
determined for a second entity of the collection, the first and ing detailed description and upon reference to the drawings
second scores determined according to the evaluation crite where:
rion; in response to recording the first and second scores, and FIG. 1 shows an exemplary hardware configuration of a
in response to receiving the evaluation indicator, update the host computer system protected from malware according to
second score according to the evaluation indicator; and in 60 Some embodiments of the present invention.
response, determine whether the second entity is malicious FIG. 2-A shows an exemplary set of software objects
according to the updated second score. including a security application executing on a host system
According to another aspect, a non-transitory computer according to some embodiments of the present invention.
readable medium stores instructions, which, when executed, FIG. 2-B shows an exemplary set of software objects,
configure at least one processor of a host system to manage a 65 including a security application executing within a virtual
collection of evaluated Software entities, wherein managing machine, in a host system configured to support virtualiza
the collection comprises: identifying a set of descendant enti tion.
 US 9,323,931 B2
3 4
FIG. 3 illustrates an exemplary hierarchy of software magnetic, optic, and semiconductor storage media (e.g. hard
objects executing on the host system at various processor drives, optical disks, flash memory, DRAM), as well as com
privilege levels, including a set of anti-malware objects munications links such as conductive cables and fiber optic
according to some embodiments of the present invention. links. According to Some embodiments, the present invention
FIG. 4 shows an exemplary sequence of steps performed by provides, interalia, computer systems comprising hardware
the entity management module of FIG.3 according to some (e.g. one or more processors) programmed to perform the
embodiments of the present invention. methods described herein, as well as computer-readable
FIG. 5 shows an exemplary scoring engine receiving a media encoding instructions to perform the methods
plurality of entity evaluation indicators determined for a soft described herein.
ware entity by a plurality of entity evaluator modules, accord 10 The following description illustrates embodiments of the
ing to Some embodiments of the present invention. invention by way of example and not necessarily by way of
FIG. 6 illustrates an exemplary execution flow of a set of limitation.
processes in a Windows.(R) environment. Solidarrows indicate FIG. 1 shows an exemplary hardware configuration of a
an exemplary execution flow in the absence of an anti-mal host system 10 performing anti-malware operations accord
ware system. Dashed arrows indicate modifications to the 15 ing to Some embodiments of the present invention. Host sys
execution flow, the modifications introduced by a plurality of tem 10 may represent a corporate computing device Such as
entity evaluators operating according to some embodiments an enterprise server, or an end-user device Such as a personal
of the present invention. computer or a Smartphone, among others. Other host systems
FIG.7 shows an exemplary sequence of steps performed by include entertainment devices such as TVs and game con
an entity evaluator module according to Some embodiments soles, or any other device having a memory and a processor
of the present invention. Supporting virtualization, and requiring malware protection.
FIG. 8 shows a plurality of exemplary entity scoring FIG. 1 shows a computer system for illustrative purposes:
objects (ESO), each ESO determined for a respective soft other client devices such as mobile telephones or tablets may
ware entity according to Some embodiments of the present have a different configuration. In some embodiments, system
invention. Exemplary data fields of an ESO include an entity 25 10 comprises a set of physical devices, including a processor
identity indicator EID, a plurality of scores S. and an aggre 12, a memory unit 14, a set of input devices 16, a set of output
gate score A determined for the respective entity, among devices 18, a set of storage devices 20, and a set of network
others. adapters 22, all connected by a set of buses 24.
FIG. 9 illustrates an exemplary set of score values, and In some embodiments, processor 12 comprises a physical
various exemplary sets of weights used by the scoring engine 30 device (e.g. multi-core integrated circuit) configured to
to score software entities according to some embodiments of execute computational and/or logical operations with a set of
the present invention. signals and/or data. In some embodiments, such logical
FIG. 10 shows an exemplary sequence of steps performed operations are delivered to processor 12 in the form of a
by the scoring engine (FIGS. 3-4) according to Some embodi sequence of processor instructions (e.g. machine code or
ments of the present invention. 35 other type of software). Memory unit 14 may comprise vola
FIG. 11 illustrates an exemplary configuration comprising tile computer-readable media (e.g. RAM) storing data/signals
a plurality of host systems connected to a security server via accessed or generated by processor 12 in the course of carry
a computer network. ing out instructions. Input devices 16 may include computer
FIG. 12 shows an exemplary anti-malware transaction keyboards, mice, and microphones, among others, including
between a host system and a security server according to 40 the respective hardware interfaces and/or adapters allowing a
Some embodiments of the present invention. user to introduce data and/or instructions into system 10.
Output devices 18 may include display devices such as moni
DETAILED DESCRIPTION OF PREFERRED tors and speakers among others, as well as hardware inter
EMBODIMENTS faces/adapters such as graphic cards, allowing system 10 to
45 communicate data to a user. In some embodiments, input
In the following description, it is understood that all recited devices 16 and output devices 18 may share a common piece
connections between structures can be direct operative con of hardware, as in the case of touch-screen devices. Storage
nections or indirect operative connections through interme devices 20 include computer-readable media enabling the
diary structures. A set of elements includes one or more non-volatile storage, reading, and writing of software instruc
elements. Any recitation of an element is understood to refer 50 tions and/or data. Exemplary storage devices 20 include mag
to at least one element. A plurality of elements includes at netic and optical disks and flash memory devices, as well as
least two elements. Unless otherwise required, any described removable media such as CD and/or DVD disks and drives.
method steps need not be necessarily performed in a particu The set of network adapters 22 enables system 10 to connect
lar illustrated order. A first element (e.g. data) derived from a to a computer network and/or to other devices/computer sys
second element encompasses a first element equal to the 55 tems. Buses 24 collectively represent the plurality of system,
second element, as well as a first element generated by pro peripheral, and chipset buses, and/or all other circuitry
cessing the second element and optionally other data. Making enabling the inter-communication of devices 12-22 of host
a determination or decision according to a parameter encom system 10. For example, buses 24 may comprise the north
passes making the determination or decision according to the bridge connecting processor 12 to memory 14, and/or the
parameter and optionally according to other data. Unless 60 Southbridge connecting processor 12 to devices 16-22,
otherwise specified, an indicator of some quantity/data may among others.
be the quantity/data itself, or an indicator different from the FIG. 2-A shows an exemplary set of software objects
quantity/data itself. Unless otherwise specified, a process executing on host system 10 in a configuration, which does
represents an instance of a computer program, wherein a not employ hardware virtualization. In some embodiments, a
computer program is a sequence of instructions determining 65 guest operating system (OS) 34 comprises software that pro
a computer system to perform a specified task. Computer vides an interface to the hardware of host system 10, and acts
readable media encompass non-transitory media Such as as a host for a set of software applications 42a-c and 44. OS
 US 9,323,931 B2
5 6
34 may comprise any widely available operating system Such system 10 instead of within VM 32. FIG. 3 is represented
as Windows.(R), MacOS(R), Linux(R), iOSR), or AndroidTM, from the perspective of processor privilege levels, also known
among others. Applications 42a-c may include word process in the art as layers or protection rings. In some embodiments,
ing, image processing, database, browser, and electronic each Such layer or protection ring is characterized by a set of
communication applications, among others. instructions, which a Software object executing at the respec
FIG. 2-B shows an exemplary set of software objects tive processor privilege level is allowed to execute. When a
executing on host system 10 in an embodiment using hard Software object attempts to execute an instruction, which is
ware virtualization. A set of guest virtual machines 32a-b are not allowed within the respective privilege level, the attempt
exposed by a hypervisor 30. Virtual machines (VM) are com may triggera processor event, such as an exception, a fault, or
monly known in the art as Software emulations of actual 10 a virtual machine exit event. In some embodiments, Switching
physical machines/computer systems, each capable of run between privilege levels may be achieved via a set of dedi
ning its own operating system and software independently of cated instructions. Such exemplary instructions include
other VMs. Hypervisor 30 comprises software allowing the SYSCALL/SYSENTER, which switch from user level to
multiplexing (sharing) by multiple virtual machines of hard kernel level, SYSRET/SYSEXIT, which switch from kernel
ware resources of host system 10, Such as processor opera 15 level to user level, VMCALL, which switches from either
tions, memory, storage, input/output, and networking user or kernel level to root level, and VMRESUME, which
devices. In some embodiments, hypervisor 30 enables mul switches from root level to either kernel or user level.
tiple virtual machines and/or operating systems (OS) to run Most components of operating system 34 execute at a
concurrently on host system 10, with various degrees of iso processor privilege level known in the art as kernel level, or
lation. To enable Such configurations, software forming part kernel mode (e.g., ring 0 on Intel platforms). An application
of hypervisor 30 may create a plurality of virtualized, i.e., 42g executes at lesser processor privilege than OS 34 (e.g.,
software-emulated devices, each virtualized device emulat ring 3, or user mode). In an embodiment Supporting virtual
ing a physical hardware device of system 10, Such as proces ization, hypervisor 30 takes control of processor 12 at the
sor 12 and memory 14, among others. Hypervisor 30 may most privileged level, also known root level or root mode
further assign a set of virtual devices to eachVM operating on 25 (e.g., ring -1 or VMXroot on Intel(R) platforms), exposing
host system 10. Thus, each VM 32a-b operates as if it pos virtual machine 32 to OS 34 and other software objects such
sesses its own set of physical devices, i.e., as a more or less as application 42g.
complete computer system. Creation and assignment of Vir In Some embodiments, parts of security application 44 may
tual devices to a virtual machine are commonly known in the execute at user-level processor privilege, i.e., same level as
art as exposing the respective VM. Examples of popular 30 application 42g. For instance, Such parts may comprise a
hypervisors include the VMware vSphereTM from VMware graphical user interface informing a user of any malware or
Inc. and the open-source Xen hypervisor, among others. security threats detected on the respective VM, and receiving
In some embodiments, hypervisor 30 includes a memory input from the user indicating, e.g., a desired configuration
introspection engine 40, configured to perform anti-malware option for application 44. Another example of a component
operations as described further below. Engine 40 may be 35 executing at user level is a user-level entity evaluator 50a,
incorporated into hypervisor 30, or may be delivered as a operating as detailed below. In some embodiments, a part of
Software component distinct and independent from hypervi user-level entity evaluator 50a may operate within security
Sor 30, but executing at Substantially similar processor privi application 44, while another part (Such as a hooking module)
lege level as hypervisor 30. A single engine 40 may be con may operate within an evaluated application, such as appli
figured to malware-protect multiple VMS executing on host 40 cation 42g. Other parts of application 44 may execute at
system 10. kernel privilege level. For instance, application 44 may install
While FIG.2-B shows just two VMs 32a-b for simplicity, an anti-malware driver 36, an entity management module 37.
host system 10 may operate a large number, e.g. hundreds, of and a scoring engine 38, all operating in kernel mode. Driver
VMs concurrently, and the number of suchVMs may change 36 provides functionality to anti-malware application 44, e.g.
during the operation of host system 10. In some embodi 45 to scan memory for malware signatures and/or to detect mal
ments, each VM 32a-b executes a guest operating system ware-indicative behavior of processes and/or other software
34a–b and/or a set of Software applications 42d-e, and 42f. objects executing on OS 34. In some embodiments, anti
respectively, concurrently and independently of other VMs malware driver 36 includes a kernel-level entity evaluator
running on host system 10. Each OS 34a–b comprises soft 50b, operating as detailed below.
ware that provides an interface to the (virtualized) hardware 50 In some embodiments, entity management module 37
of the respective VM32a-b, and acts as a host for computing manages a collection of Software entities executing within
applications executing on the respective OS. host system 10 (or VM 32). In some embodiments, the col
In some embodiments, a security application 44 is config lection comprises all entities being evaluated for malware by
ured to performanti-malware operations as detailed below, to entity evaluation modules such as 55a-b. To manage the col
protect host system 10 from malware. In the example of FIG. 55 lection, module 37 may add and/or remove entities from the
2-B, an instance of application 44 may execute on each VM collection in response to detecting the occurrence of life
32a-b, each Such instance configured to protect the respective cycle events, such as entity launch and/or termination events,
virtual machine. Security application 44 may be a standalone as shown in mode detail below. Module 37 may further deter
program, or may form part of a Software Suite comprising, mine inter-entity relationships, such as determine child enti
among others, anti-malware, anti-spam, and anti-spyware 60 ties (e.g., child processes) of a parent entity, and/or determine
components. whether a selected entity has injected a software object, such
FIG. 3 illustrates a hierarchy of software objects executing as a library, into another entity, or whether the selected entity
on host system 10 according to Some embodiments of the is the target of injection by another software entity. A child
present invention. FIG. 3 shows an exemplary embodiment entity is an executable entity created by another executable
configured to execute in a virtualization environment; it may 65 entity called the parent entity, the child entity executing inde
be clear to a person skilled in the art that the illustrated pendently from the parent entity. Exemplary child entities are
embodiment may be modified to execute directly on host child processes, for instance created via the CreateProcess
 US 9,323,931 B2
7 8
function of the Windows(R OS, or via the fork mechanism in entity is malicious according to the respective data. In some
Linux R. Code injection is a generic term used in the art to embodiments, Software entities analyzed by scoring engine
indicate a family of methods of introducing a sequence of 38 include, among others, executable objects such as pro
code. Such as a dynamic-link library (DLL), into the memory cesses and execution threads. A process is an instance of a
space of an existing process, to alter the original functionality computer program, Such as an application or a part of an
of the respective process. To perform tasks Such as detecting operating system, and is characterized by having at least an
the launch of a process and/or detecting code injection, mod execution thread and a section of virtual memory assigned to
ule 37 may employ any method known in the art, such as it by the operating system, the respective section comprising
calling or hooking certain OS functions. For instance, in a executable code. In some embodiments, evaluated Software
system running a Windows(R OS, module 37 may register a 10 entities may vary Substantially in Scope and complexity, for
PsSetCreateProcessNotifyRoutine callback to detect the instance from individual threads, to individual applications,
launch of a new process, and/or hook the CreateRemoteTh to whole instances of operating systems and/or virtual
read function to detect execution of injected code. machines.
FIG. 4 shows an exemplary sequence of steps performed by FIG. 5 shows an exemplary scoring engine 38 receiving a
entity management module 37 according to Some embodi 15 plurality of evaluation indicators 52a-d, each indicator 52a-d
ments of the present invention. In a sequence of steps 250 determined by an entity evaluator. In FIG. 5, such evaluators
252, module 37 intercepts an entity life-cycle event, using, for include user-level entity evaluator 50a, kernel-level entity
instance, the methods described above. When such an event evaluator 50b, and a system call evaluator 50c, among others.
has occurred, a step 254 identifies the entity triggering the Each Such evaluator module may execute independently of
respective event. Step 258 may include determining a unique other evaluators, and each may determine a plurality of dis
entity identification indicator (EID) of the respective entity: tinct entity evaluation indicators of the evaluated software
Such an indicator may be used in scoring the respective entity, entity. In systems implementing hardware virtualization,
as shown further below. A step 256 determines whether the Some evaluation indicators, such as indicators 52a-c in FIG.
event comprises a launch of a new entity (e.g., a new process), 5, are determined by components executing within VM 32.
and when no, module 37 advances to a step 260. When the 25 while otherevaluation indicators, such as 52d, are determined
event comprises a launch, in a step 258, module 37 may add by components executing outside VM 32 (for instance, by
the triggering entity to the collection of evaluated entities. memory introspection engine 40). In some embodiments,
Step 260 comprises determining whether the event comprises each evaluation indicator 52a-d comprises an entity identifi
a parent entity spawning a child entity, and when no, module cation indicator, allowing engine 38 to uniquely associate the
37 may advance to step 264. When yes, in a step 262, module 30 respective evaluation indicator to the software entity for
37 may add the respective child entity to the collection of which it was determined.
evaluated entities. Step 262 may further include determining Some evaluation indicators may be malware-indicative,
an EID of the child entity, and registering a relation between i.e., may indicate that the evaluated entity is malicious. Some
the triggering entity and the child entity as a filiation (parent evaluation indicators may not be malware-indicative them
child) relation. 35 selves, but may indicate maliciousness when combined with
In some embodiments, step 264 determines whether the other evaluation indicators. Each evaluation indicator 52a-d
event comprises an injection of code, and whenno, module 37 may be determined according to a distinct method and/or
may advance to a step 268. When yes, module 37 may identify criterion. Exemplary evaluation criteria include behavioral
a source entity and a target entity of the code injection, criteria, Such as determining whether the evaluated entity
wherein the source entity injects code into the target entity. In 40 performs a certain action, Such as writing to a disk file, editing
a step 266, module 37 may register a relation of the code a system register key of VM32, or writing to a memory page
injection type between the source entity and the target entity. belonging to a protected Software object. Another exemplary
In step 268, entity management module 37 determines criterion may include determining whether a section of
whether the event comprises the termination of the triggering memory belonging to the evaluated entity contains a mal
entity, and when no, module 37 returns to step 250. In some 45 ware-indicative signature.
embodiments, an entity is considered terminated when all To illustrate the operation of entity evaluators 50a-c, FIG.
components of the respective entity have finished execution. 6 shows an exemplary execution flow of a set of software
For instance, a process is terminated when all the threads of entities 70a-baccording to some embodiments of the present
the respective process have finished execution. When the invention. For simplicity, the chosen entities 70a-b are pro
event comprised the termination of the triggering entity, in a 50 cesses executing in an instance of a Windows(R OS; similar
step 270, module 37 may determine a set of descendant enti diagrams may be rendered for other operating systems such
ties of the triggering entity. In some embodiments, descen as Linux, for instance. Solid arrows represent the execution
dant entities of the triggering entity include children entities flow in the absence of entity evaluators (e.g., in the absence of
of the respective entity, as well as children entities of the security application 44). Dashed arrows represent modifica
children entities, over multiple generations. In some embodi 55 tions to the flow due to the presence of entity evaluators 50a-c
ments, descendant entities may include target entities com executing according to Some embodiments of the present
prising code injected by the triggering entity, as well as enti invention.
ties targeted by the targeted entities, recursively. In a step 272, Process 70a loads a plurality of dynamic-linked libraries
module 37 may determine whether all entities of the set of (DLLs) 72a-c; in the example of FIG. 6, DLL 72c is injected
descendant entities are terminated, and when no, execution 60 into process 70a by (possibly malicious) process 70b. When
returns to step 250. When all descendants are terminated, in a process 70a (or one of its loaded DLLs) executes an instruc
step 274, entity management module 37 may remove the tion calling for some system functionality, e.g. to write some
triggering entity from the collection of evaluated entities. thing to a disk file, or to edit a registry key, the respective
In some embodiments, scoring engine 38 is configured to instruction calls a user-mode API such as KERNEL32.DLL
receive data from a plurality of entity evaluator modules, such 65 or NTDLL.DLL. In the example of FIG. 6, the respective
as evaluators 50a-b, the data determined for an evaluated user-mode API call is intercepted and analyzed by user-level
software entity, and to determine whether the respective behavioral filter 50a. Such interceptions may be achieved by
 US 9,323,931 B2
10
a method such as DLL injection or hooking, among others. actual data may be transferred, for instance, through the
Hooking is a generic term used in the art for a method of shared memory section described above.
intercepting function calls, or messages, or events passed FIG.7 shows an exemplary sequence of steps performed by
between software components. One exemplary hooking an entity evaluator, Such as evaluators 50a-C and/or memory
method comprises altering the entry point of a target function, introspection engine 40 in FIGS. 4-5, according to some
by inserting an instruction redirecting execution to a second embodiments of the present invention. In a sequence of steps
function. Following such hooking, the second function may 302-304, the entity evaluator waits for the occurrence of a
be executed instead, or before, the target function. In the trigger event within host system 10 and/or virtual machine32.
example of FIG. 6, anti-malware driver 36 may hook into Exemplary trigger events include, among others, a Software
certain functions of KERNEL32.DLL or NTDLL.DLL, to 10 entity performing a certain behavior, Such as issuing a par
instruct the respective functions to redirect execution to filter ticular processor instruction, attempting to use a particular
50a. Thus, filter 50a may detect that process 70a is attempting piece of hardware such as storage devices 20 or network
to perform a certain behavior, identified according to the adapter(s) 22, or attempting to write to a protected memory
hooked function. When filter 50a detects such behavior, filter page. For instance, a trigger event for evaluator 50c may
50 may formulate evaluation indicator 52a and transmit indi 15 include a software entity issuing a system call (e.g.,
cator 52a to scoring engine 38 (see e.g., FIG. 5). SYSENTER). Another example of a trigger event for evalu
In a typical flow of execution, the user-mode API function ator 50d may include an application calling a function of the
called by entity 70a may request service from the operating UrlDownloadTofile API. To detect the occurrence of a trig
system’s kernel. In some embodiments, such operations are ger event, the respective entity evaluator may use any method
carried out by issuing a system call, such as SYSCALL and known in the art, such as code injection and MSR hooking,
SYSENTER on x86 platforms. In the example of FIG. 6, such among others. Some examples of trigger event interception
system calls are intercepted by system call evaluator 50c. In are described above, in relation to FIG. 6.
Some embodiments, such interception comprises, for When a trigger event is detected, in a step 306, the entity
instance, modifying a system call handler routine by chang evaluator may identify the Software entity (e.g., process)
ing a value stored in a model-specific register (MSR) of 25 causing the respective trigger event. In some embodiments,
processor 12, which effectively redirects execution to filter the entity evaluator may determine the identity of the software
50c. Such techniques are known in the art as MSR hooking, entity from a data structure used by OS 34 to represent each
and may allow system call evaluator 50c to detect that the process and/or thread currently in execution. For instance, in
evaluated process is attempting to perform certain system Windows, each process is represented as an executive process
calls. When Such system calls are intercepted, system call 30 block (EPROCESS), which comprises, among others,
filter 50c may formulate entity evaluation indicator 52c and handles to each of the threads of the respective process, and a
transmit indicator 52c to scoring engine 38. unique process ID allowing OS 34 to identify the respective
Following the system call, control of the processor is typi process from a plurality of executing processes. Similar pro
cally turned over to the kernel of OS 34. In some embodi cess/thread representations are available for other OSs, such
ments, kernel-level entity evaluator 50b is configured to inter 35 as Linux.
cept certain operations of the OS kernel, and therefore In a step 308, the entity evaluator may formulate an evalu
determine that the evaluated process is attempting to perform ation indicator, including an identifier (e.g., process ID) of the
certain operations, which may be malicious. To intercept Such respective software entity and an indicator of the kind of
operations, some embodiments may employ a set of filtering action/event performed by the respective software entity and
mechanisms built into and exposed by OS34. For example, in 40 intercepted in steps 302-304. In some embodiments, the
a Windows OS, FltRegisterFilter may be used to intercept entity evaluator may determine a type of action and/or behav
operations like creating, opening, writing to, and deleting a ior of the respective software entity, from parameters of the
file. In another example, evaluator 50b may use ObRegister intercepted trigger event. In an example of operation, when a
Callback to intercept create or duplicate object-handle opera process attempts to download a file from the Internet, user
tions, or PsSetCreateProcessNotifyRoutine to intercept the 45 level entity evaluator 50a may intercept the attempt. Beside
creation of new processes. In yet another example, Windows identifying which process is performing the action, evaluator
registry operations such as creating and setting registry keyS/ 50a may also determine the type of action (downloading a
values may be intercepted using CmRegisterCallbackEx. file), the IP address that the file is downloaded from, and the
Similar filtering mechanisms are known in the art for other disk location of the downloaded file, among others. Such data
operating systems such as Linux R. When kernel-mode entity 50 may be selectively incorporated into the evaluation indicator,
evaluator 50b intercepts such operations, evaluator 50b may allowing scoring engine 38 to determine that entity X has
formulate entity evaluation indicator 52b and transmit indi performed action Y, with parameters Z. In a step 310, the
cator 52b to scoring engine 38. entity evaluator transmits the evaluation indicator to scoring
To transmit data, Such as entity evaluation indicators 52a-c, engine 38.
from evaluators 50a-c to scoring engine38, a person skilled in 55 In some embodiments, scoring engine 38 and/or entity
the art may employ any inter-process communication management module 37 maintain a centralized knowledge
method. For instance, to communicate between user-mode base of evaluated Software entities, such as processes and
and kernel-mode components, evaluators 50a-c and engine threads executing on host system 10 (orVM32). FIG.8 shows
38 may be configured to use a shared section of memory. a set of evaluated entities 70c-e, each represented as an exem
When data exchange is needed between components execut 60 plary entity scoring object (ESO) 74a-c, respectively. Each
ing within VM 32, and components executing outside the ESO comprises a plurality of data fields, some of which are
respectiveVM. Such communication may be carried out using illustrated in FIG. 8. Such fields may include a unique entity
any method known in the art of virtualization. For instance, to identifier (EID) 76a, a plurality of evaluation scores 76b, and
transmit evaluation indicator 52d from memory introspection an aggregate score 76d. In some embodiments, evaluation
engine 40 to scoring engine 38, Some embodiments use an 65 scores 76b are determined by engine 38 according to evalu
interrupt injection mechanism to signal to engine 38 that data ation indicators 52a-d received from individual entity evalu
is being transmitted from outside the respective VM. The ators. Each Such score may be determined according to an
 US 9,323,931 B2
11 12
evaluation criterion, identified by indicators 76c. In some from the first) applicable to web browsers (e.g., FirefoxR) and
embodiments, evaluation scores 76b have a one-to-one cor MS Internet Explorer(R), and a third weight value applicable
respondence with evaluation criteria 76c, so that each score is to file manager applications (e.g., Windows ExplorerR). Dis
attributed according to the respective criterion. For instance, tinguishing among different categories of entities may be
a particular criterion C may comprise determining whether 5 useful, since Some evaluation criteria may be more malware
the evaluated entity downloads a file from a computer net indicative for one category of entities than for others. More
work Such as the Internet. In one Such example, the respective generally, each scoring weight may be indexed by a tuple
score S may be awarded only if the evaluated entity attempts <C.E. . . . D, wherein C, denotes a particular evaluation
a download. criterion, and wherein Edenotes aparticular evaluated entity.
In some embodiments, ESO 74a may further comprise a set 10 The actual data format for storing and accessing scoring
of flags 76e. Some flags 76e may be binary indicators (e.g., weights 82a-e may vary among embodiments. Weights 82a-e
0/1, yes/no). In one such example, a flag indicates whether the may be stored as matrices, lists, relational databases (RDB),
respective evaluated entity El Satisfies a particular evaluation or extensible markup language (XML) structures, among oth
criterion (e.g., whether E is an executable file downloaded ers. An exemplary use of weights for scoring will be discussed
from the Internet, whether E runs in command line mode, 15 below.
etc.). Another exemplary flag is indicative of a classification Score values 80 and/or weights 82a-e are predetermined,
of entity E, e.g., an indicator that E belongs to a particular for instance by a human operator. In some embodiments, such
category of objects, such as Trojan malware, browser objects, values may change in time, and may be adjusted to optimize
PDF reader applications, etc. An exemplary use of flags com malware detection. Updated score values and/or weight val
prises a situation wherein an update of an evaluation score S, ues may be delivered to host system 10 as periodic and/or
of entity E triggers an update of another evaluation score S, on-demand Software updates from a security server (see
of E (see below). Flags may be used to turn Such co-update below, in relation to FIGS. 10-11).
mechanisms on and off. For instance, it may be known that FIG.10 shows an exemplary sequence of steps executed by
when El Satisfies evaluation criterion C, (e.g., if the entity scoring engine 38 according to Some embodiments of the
performs a particular action), entity E is also likely to satisfy 25 present invention. In a step 302, engine 38 receives an entity
criterion C. Therefore, a flag F, indicating the connection evaluation indicator from an entity evaluator, for instance one
<C,C> may be set for entity E, triggering an update of score of evaluators 50a-c in FIG. 5. In some embodiments imple
S, when score S, is updated. menting hardware virtualization, engine 38 may receive the
ESO 74a may further include a termination indicator 76f. respective entity evaluation indicator from a component
indicating whether the respective entity is currently active or 30 executing outside of the respective virtual machine (e.g.,
terminated. Such termination indicators may allow scoring memory introspection engine 40 in FIG. 5). In a step 304,
engine 38 to keep records of and/or update scores of termi engine 38 may identify the software entity for which the
nated entities. ESO 74a may further include a set of identifiers respective entity evaluation indicator was determined, e.g.,
of software entities related to the respective entity E. according to an entity ID embedded in the respective evalu
Examples of Such related Software entities may comprise a 35 ation indicator (see above, in relation to FIG. 7).
parent entity of E, denoted by identifier 76g, and a set of Next, scoring engine 38 performs a block of steps 318-332,
children entities of E, denoted by identifier 76.h. ESO 74a for the entity E identified in step 304, as well as for other
may further comprise a set of indicators of injection target entities related to E. Such related entities may include parent
entities (items 76.j), identifying software entities into which and child entities of E, injection target entities into which E
E has injected code, and further a set of indicators of injec 40 has injected code, and injection Source entities which have
tion source entities (items 76 k), identifying software entities injected code into E, among others. In this manner, each time
which have injected code into E. engine 38 receives an evaluation indicator (indicating, for
In some embodiments, scoring of evaluated Software enti instance, that entity E has performed a particular action),
ties proceeds according to a set of score values and further block 318-332 may execute several times, updating not only
according to additional parameters. FIG. 9 illustrates such 45 the evaluation scores of entity E, but also the evaluation
data, wherein a set of score values is denoted by item 80. scores of entities related to E. In some embodiments, block
Score values are indexed by their corresponding evaluation 318-332 is executed once for E and once for each entity E*
criteria C. . . . C. Each Such value may represent, for related to E. In alternative embodiments, block 318-332 is
instance, a predetermined number of points that an evaluated executed recursively, until Some convergence criterion is sat
entity receives if it satisfies the respective evaluation criterion 50 isfied. An exemplary convergence criterion comprises verify
(e.g., if it downloads a file from the Internet, if it writes to a ing whether evaluation scores of E and/or E* change between
MS Word(R) document, etc.). successive executions of block 318-332, and exiting when
Exemplary parameters controlling scoring include a set of there is no such change. In the exemplary algorithm of FIG.
initialization weights 82a, a set of propagation weights 82b, a 10, the variable X is used to indicate the entity currently
set of new instance weights 82c, a set of exception weights 55 undergoing score updates. In a step 316, X is set to the entity
82d, and a set of flag-induced weights 82e. Weights 82a-e are E identified in step 304.
indexed by the evaluation criteria C. . . . C. Some type of In a step 318, engine 38 updates evaluation scores of entity
weights are in a one-to-one correspondence with evaluation X (e.g., entities 76b in FIG. 8). In some embodiments, updat
criteria, so that there is one weight value w, for each C. Other ing an evaluation score comprises replacing a recorded value
types of weights are in a one-to-many correspondence with 60 of the respective evaluation score with a new value:
evaluation criteria. One such example is exception weights
82d in FIG.9, wherein there may be a plurality of weights wi Six >S.--AS, 1.
corresponding to a particular evaluation criterion C. Weights wherein S. denotes the evaluation score determined for
may be grouped by classes or categories of entities, as illus entity X according to evaluation criterion C, and wherein
trated by the example of FIG. 9; for instance, there may be a 65 AS denotes an increment, which may be positive or negative
first weight value applicable to word processing applications (in Some embodiments, evaluation scores may decrease upon
(e.g., MS Word(R), a second weight value (possibly distinct update).
 US 9,323,931 B2
13 14
In some embodiments, the score increment AS is deter In some embodiments, updating an evaluation score S.
mined by scoring engine 38 according to the evaluation indi may trigger an update of a distinct evaluation score S of the
cator received in step 312. The respective indicator may respective entity. For instance,
include a score and/or indicate an evaluation criterion used in
determining the respective indicator. In some embodiments, S->S+V triggers S->S+Ff. V. 4
scoring engine 38 determines the score increment AS accord wherein F is a flag set for entity X (see e.g., items 76e in
ing to score value 80 corresponding to the respective evalua FIG. 8), the flag indicative of a connection between evalua
tion criterion C (see FIG. 9), for instance: tion criteria C and C, and wherein f is a flag-induced
ASFV 2 10
weight (see e.g., item 82e in FIG. 9), indicating a strength
with which the update of S influences the update of S, of
wherein V denotes score value 80 assigned to criterion C. In entity X.
one such example, wherein criterion C comprises determin In a step 320, scoring engine 38 may update flags of entity
ing whether entity X downloads an object from the network, X (see discussion on flags above, in relation to FIG. 8),
and wherein V 20, the evaluation score S, will be according to the evaluation indicator received in step 312.
increased by 20 points every time entity Xperforms a down 15
Flags may be set to activate and/or de-activate score co
load. In some embodiments, AS eV, whereine is a binary updating mechanisms, such as described above, in relation to
exception weight (see e.g. items 82d in FIG.9), forcing score Eq. 4. In one such example, an evaluated entity might be
S. to be updated only for a subset of evaluated entities. Such identified as being a web browser application according to the
exception weights are useful, for example to distinguish evaluation indicator (step 312); such identification should
between various types of evaluated entities. For instance, a indicate to scoring engine 38 not to score the respective entity
browser should be allowed to access an unlimited number of
IP addresses without rising Suspicion of malware; an evalua for future downloads from the Internet. This may be achieved
tion criterion including detecting Internet access may effec by setting the value of a specific flag F to 0 for the respective
entity, wherein flag Findicates to scoring engine 38 to update
tively be switched off for browser objects by setting the an evaluation score of the respective entity, when the entity
exception weight to 0 for entities of the browser type, while 25
downloads an object from the Internet.
keeping it active (e=1) for other types of entities. In a step 322, scoring engine 38 may determine an aggre
In some embodiments, the score increment AS used in gate score of entity X by combining individual evaluation
updating the evaluation score of entity X is determined scores determined for the respective process, for instance as a
according to an evaluation score determined for an entity X* SU
related to X, i.e., scores may propagate from one entity to a 30
related entity, such as from a child to a parent, or from an
injection target to the source of the injection. In one such
example, an action performed by a child process may trigger
an update not only of a score of the entity performing the
action (the child process), but also of a score of the parent 35
process of the respective child process. Such score updates In a step 324, engine 38 may compare the aggregate score
may comprise computing the score increment according to: to a predetermined threshold. When the aggregate score does
not exceed the threshold, scoring engine 38 may proceed to a
AS =w.S., 3) step 326 described below. In some embodiments, the thresh
wherein w denotes a numerical, criterion-specific weight, 40 old may be set to a value determined according to a user input.
indicating a strength with which the score of entity X* affects Such threshold values may reflect the respective user's secu
the score of entity X. Weights w may include propagation rity preferences. For instance, when the user opts for tight
weights 82b (FIG.9). Some embodiments distinguish among security, the threshold may be set to a relatively low value:
a variety of Such propagation weights, for instance weights when the user prefers a more tolerant security setting, the
used to propagate scores from a child entity to a parent entity 45 threshold may be set to a relatively high value. In some
may differ in value from weights used to propagate scores embodiments, the threshold value may be received from a
from the parent entity to the child entity. Similarly, weights remote security server, as described below in relation to
used to propagate scores from the child entity to the parent FIGS 10-11.
entity may differ in value from weights used to propagate In some embodiments, in steps 322-324, scoring engine 38
scores from an entity targeted for code injection to the entity 50 may determine a plurality of aggregate scores, and compare
performing the code injection. In some embodiments, scores each aggregate score to a (possibly distinct) threshold. Each
may propagate from active entities to terminated entities. For Such aggregate score may be determined according to a dis
instance, an action of a child process may increment the score tinct Subset of evaluation scores. In an exemplary embodi
of the parent process, even when the parent process is termi ment, each Such Subset of scores, and their corresponding
nated. 55 Subset of evaluation criteria, may represent a particular class
In some embodiments, entity X* in Eqn. 3 is another or type of malware (e.g., Trojans, rootkits, etc.). This may
instance of entity X. For example, X and X* may be copies of allow engine 38 to perform a classification of the detected
the same process or thread, executing concurrently. In Such malware. In another embodiment, scoring engine 38 employs
cases, weight W may be a new instance weight (e.g., item 82c a plurality of threshold values in order to classify the execu
in FIG.9), oran initialization weight (e.g., item 82a). In some 60 tion entities according to various degrees of maliciousness
embodiments, when a new instance X" of the entity X is (e.g. clean, Suspicious, dangerous and critical).
launched, engine 38 may update Some or all evaluation scores When the aggregate score exceeds the threshold, in a step
of the existing entity X, using new instance weights w to 326, engine 38 may decide that the evaluated process is mali
propagate scores from X to X'. Similarly, when X" is cious, and may take anti-malware action. In some embodi
launched, engine 38 may update Some or all evaluation scores 65 ments, such anti-malware action may include, among others,
of X", using initialization weights w propagate scores from terminating the evaluated process, quarantining the evaluated
the already executing entity X to the new entity X". process, and removing or disabling a resource (Such as a file
 US 9,323,931 B2
15 16
or a section of memory) of the evaluated process. In some executing on host system 10, and/or aggregate scores deter
embodiments, anti-malware action may further comprise mined by scoring engine 38, among others. Security report 80
alerting a user of host system 10, and/or alerting a system may also comprise data identifying the respective system 10
administrator, for instance by sending a message to the sys and evaluated entities (e.g., entity IDs, names, paths, hashes,
tem administrator over a computer network connected to host or other kinds of entity identifiers), as well as indicators
system 10 via network adapter(s) 22. In some embodiments, associating an entity evaluation indicator/score to the host
anti-malware action may also comprise sending a security system and entity for which it was determined. In some
report to a remote security server, as described below in embodiments, report 80 may further comprise statistical and/
relation to FIGS. 10-11. or behavioral data regarding entities executing on host system
In a sequence of steps 328-330, engine 38 may identify an 10 10. System 10 may be configured to send report 80 upon
entity X* related to X, wherein scores of X* need updating detection of malware, and/or according to a schedule (e.g.,
following the current score updates of X. For instance, X* every few minutes, every hour, etc.).
may be a parent or a child entity of X. In some embodiments, In some embodiments, security settings 82 may include
entities X* may be identified according to fields 76g-k of the operational parameters of entity evaluators (e.g., parameters
ESO of entity X (see, e.g., FIG. 8). When no such entities X* 15 offilters 50a-c in FIG.5), and/or parameters of scoring engine
exist, or when all such entities X* have already been consid 38. Exemplary parameters of engine 38 include the threshold
ered for score updates, engine 38 returns to step 312. When for deciding whether an evaluated process is malicious, as
there is at least an entity X*, in a step 332 scoring engine well as score values 80 and weights 82a-e, among others.
makes X* the current entity and returns to step 318. In some embodiments, server 110 runs an optimization
The exemplary scoring engine 38 depicted in FIGS. 3-4 algorithm to dynamically adjust Such parameters to maximize
operates within VM32 at OS processor privilege level (e.g., malware-detection performance, for instance to increase
kernel mode). In alternative embodiments, scoring engine 38 detection rate while minimizing false positives. Optimization
may execute within VM32 in user mode, or even outside VM algorithms may receive statistical and/or behavioral data
32, at the processor privilege level of hypervisor 30. about various entities executing on the plurality of host sys
In some embodiments, introspection engine 40 executes 25 tems 10a-c, including entity evaluation indicators/scores
substantially at the same privilege level as hypervisor 30, and reported to scoring engine 38 by various entity evaluators,
is configured to perform introspection of virtual machines and determine optimal values for the parameters. The values
such as VM 32 (FIG. 3). Introspection of a VM, or of a are then transmitted to the respective host systems via net
Software entity executing on the respective VM, may com work 26.
prise analyzing a behavior of the Software entity, determining 30 In one such example of optimization, changing score val
and/or accessing memory addresses of Such entities, restrict ues 80 may effectively change the relevance of the respective
ing access of certain processes to a content of memory located evaluation criteria, relative to each other. Malware threats
at Such addresses, analyzing such content, and determining typically occur in waves, in which a great number of com
evaluation indicators of the respective entities (e.g., indicator puter systems worldwide are affected by the same malware
52d in FIG. 5), among others. 35 agent in a short time interval. By receiving security reports 80
In some embodiments, host system 10 may be configured in real time from a plurality of host systems, security server
to exchange security information, Such as details about mal 110 may be kept up to date with the current malware threats,
ware detection events, with a remote security server. FIG. 11 and may promptly deliver optimal security settingS 82 to the
illustrates such an exemplary configuration, in which a plu respective host systems, settings 82 including, for instance, a
rality of host systems 10a-c, such as system 10 discussed 40 set of score values 80 optimized for detecting the current
above, are connected to a security server 110 via a computer malware threats.
network 26. In an exemplary embodiment, host systems The exemplary systems and methods described above
10a-care individual computers used by employees of a cor allow protecting a host system, such as a computer system,
poration, while security server 110 may comprise a computer from malware such as viruses, Trojans, and spyware. For each
system configured by a network administrator of the respec 45 of a plurality of executable entities, such as processes and
tive corporation to monitor malware threats or security events threads currently executing on the host system, a scoring
occurring on systems 10a-c. In another embodiment, for engine records a plurality of evaluation scores, each score
instance in an Infrastructure-as-a-service (IAAS) system determined according to a distinct evaluation criterion. In
wherein each host system 10a-c is a server hosting tens or Some embodiments, evaluated Software entities may vary
hundreds of virtual machines, security server 110 may com 50 Substantially in scope and complexity, for instance from indi
prise a computer system configured to manage anti-malware vidual execution threads, to individual applications, to whole
operations for all such VMs from a central location. In yet instances of operating systems and/or virtual machines.
another embodiment, security server 110 may comprise a Every time a monitored entity satisfies an evaluation crite
computer system configured by a provider of anti-malware rion (e.g., performs an action), the respective score of the
Software (e.g., the provider of security application 44, among 55 entity is updated. Updating a score of a target entity may
others), to receive statistical and/or behavioral data about trigger score updates of other entities related to the target
malware detected on various systems around network 26. entity. Such related entities include, among others, children of
Network 26 may include a wide-area network such as the the target entity, the parent of the target entity, entities into
Internet, while parts of network 26 may include local area which the target entity has injected code, and entities which
networks (LAN). 60 have injected code into the target entity.
FIG. 12 shows an exemplary data exchange between host Conventional anti-malware systems typically score each
system 10 and security server 110 in an embodiment as shown entity separately from other entities. Some malware may try
in FIG. 11. Host system 10 may be configured to send a to evade detection by dividing malicious activities among
security report 80 to server 110, and to receive a set of security several distinct agents, such as children processes of a mali
settings 82 from server 110. In some embodiments, security 65 cious process, so that none of the individual agents performs
report 80 comprises entity evaluation indicators 52a-d and/or sufficient malware-indicative activity to be detected. In con
scores determined by entity evaluators 50a-c and/or 40, trast, some embodiments of the present invention propagate
 US 9,323,931 B2
17 18
scores from one entity to other related entities, thus corrobo What is claimed is:
rating malware-indicative data across related entities. Score 1. A host system comprising a memory unit storing instruc
propagation may ensure that at least one of the agents tions which, when executed by at least one hardware proces
involved in malicious activities is detected. sor of the host system, cause the host system to forman entity
In one exemplary evasion strategy, a malware agent may management module, an entity evaluator, and a scoring
spawn a plurality of child processes and quit. Malicious engine, wherein:
activities may be divided among child processes, such that the the entity management module is configured to manage a
actions of no individual child may triggera malware alarm on collection of evaluated software entities, wherein man
their own. In some embodiments of the present invention, aging the collection comprises:
scores may propagate from one entity to another, even when 10
identifying a set of descendant entities of a first entity of
the latter is terminated. Such a configuration may detect the the collection;
parent process as malicious, even if it may fail to detect determining whether the first entity is terminated;
maliciousness of the child processes. Some embodiments in response, when the first entity is terminated, deter
maintain a list of entities currently under evaluation; the list
may include both active and terminated entities. An entity 15 mining whether all members of the set of descendant
may be taken off the list only when all descendants of the entities are terminated; and
respective entity are terminated. in response, when all members of the set of descendant
In conventional anti-malware systems, only one score is entities are terminated, removing the first entity from
typically recorded for each entity. By keeping a plurality of the collection;
per-entity scores, each computed according to its distinct the entity evaluator is configured to:
criterion, Some embodiments of the present invention allow evaluate the first entity according to an evaluation crite
scores to be propagated among related entities on a per rion; and
criterion basis. Such scores may either increase or decrease in response, when the first entity satisfies the evaluation
upon propagation, allowing a more precise assessment of criterion, transmit an evaluation indicator to the scor
maliciousness throughout the life cycle of each entity, with 25 ing engine; and
fewer false positive detections. In some embodiments, the the scoring engine is configured to:
extent to which scores of one entity affect scores of a related record a first score determined for the first entity and a
entity is adjustable via a numerical propagation weight. Such second score determined for a second entity of the
weights may differ from one entity to another and/or from one collection, the first and second scores determined
evaluation criterion to another, allowing a flexible and precise 30
according to the evaluation criterion;
tuning of score propagation. Weight values may be deter in response to recording the first and second scores, and
mined by human operators and/or be subject to automated in response to receiving the evaluation indicator,
optimization aimed at improving malware detection perfor update the second score according to the evaluation
aCC.
Some conventional anti-malware systems determine 35 indicator,
whether an evaluated entity is malicious by determining in response, determine whether the secondentity is mali
whether the respective entity performs a malware-indicative cious according to the updated second score;
behavior, and/or whether the entity has malware-indicative in response to receiving the evaluation indicator, update
features, such as a malware-indicative sequence of code. In the first score according to the evaluation indicator;
contrast, in some embodiments of the present invention, 40 and
entity evaluation criteria are not necessarily malware-indica in response, determine whether the first entity is mali
tive on their own. For instance, some criteria include deter cious according to the updated first score.
mining whether an entity performs benign actions such as 2. The host system of claim 1, wherein the first entity is a
opening a file or accessing an IP address. Nevertheless, Such child of the second entity.
actions may be malicious when combined with other actions, 45 3. The host system of claim 1, wherein the second entity is
which themselves may not be malware-indicative on their a child of the first entity.
own. By monitoring a wide variety of entity behaviors and/or 4. The host system of claim 1, wherein the first entity
features, Subsequently recording a large number (possibly comprises a section of code injected by the second entity.
hundreds) of evaluation scores, and aggregating Such scores 5. The host system of claim 1, wherein the second entity
in a per-entity fashion, some embodiments of the present 50 comprises a section of code injected by the first entity.
invention may increase detection rates, while minimizing 6. The host system of claim 1, wherein updating the second
false positives. score comprises changing the second score by an amount
Some embodiments of the present invention may protect a determined according to wS, wherein S is the first score, and
virtualized environment. In an embodiment configured to wherein w is a numerical weight.
Support virtualization, some components of the present 55 7. The host system of claim 1, wherein managing the col
invention may execute within a virtual machine, whereas lection of evaluated software entities further comprises:
others may execute outside the respective virtual machine, for intercepting a launch of a new software entity; and
instance at the level of a hypervisor exposing the respective in response, adding the new software entity to the collec
virtual machine. Such components executing at hypervisor tion.
level may be configured to perform anti-malware operations 60 8. A non-transitory computer-readable medium storing
for a plurality of virtual machines executing concurrently on instructions, which, when executed, configure at least one
the respective host system. processor of a host system to:
It will be clear to one skilled in the art that the above manage a collection of evaluated Software entities, wherein
embodiments may be altered in many ways without departing managing the collection comprises:
from the scope of the invention. Accordingly, the scope of the 65 identifying a set of descendant entities of a first entity of
invention should be determined by the following claims and the collection;
their legal equivalents. determining whether the first entity is terminated;
 US 9,323,931 B2
19 20
in response, when the first entity is terminated, deter 14. The computer-readable medium of claim 8, wherein
mining whether all members of the set of descendant managing the collection of evaluated software entities further
entities are terminated; and comprises:
in response, when all members of the set of descendant intercepting a launch of a new software entity; and
entities are terminated, removing the first entity from 5 in response, adding the new software entity to the collec
tion.
the collection; 15. A method comprising employing at least one processor
record a first score determined for the first entity and a of a host system to:
second score determined for a second entity of the col manage a collection of evaluated software entities, wherein
lection, the first and second scores determined according 10 managing the collection comprises:
to an evaluation criterion; identifying a set of descendant entities of a first entity of
in response to recording the first and second scores, evalu the collection;
ate the first entity according to the evaluation criterion; determining whether the first entity is terminated:
in response to evaluating the first entity, when the first in response, when the first entity is terminated, deter
entity satisfies the evaluation criterion, update the sec 15 mining whether all members of the set of descendant
ond score; entities are terminated; and
in response to updating the second score, determine in response, when all members of the set of descendant
whether the second entity is malicious according to the entities are terminated, removing the first entity from
updated second score;
in response to evaluating the first entity, when the first the collection;
entity satisfies the evaluation criterion, update the first record a first score determined for the first entity and a
score; and second score determined for a second entity of the col
in response, determine whether the first entity is malicious lection, the first and second scores determined according
according to the updated first score. to an evaluation criterion;
9. The computer-readable medium of claim 8, wherein the in response to recording the first and second scores, evalu
first entity is a child of the second entity. 25 ate the first entity according to the evaluation criterion;
10. The computer-readable medium of claim8, wherein the in response to evaluating the first entity, when the first
second entity is a child of the first entity. entity satisfies the evaluation criterion, update the sec
11. The computer-readable medium of claim8, wherein the ond score;
first entity comprises a section of code injected by the second in response to updating the second score, determine
entity. 30 whether the second entity is malicious according to the
12. The computer-readable medium of claim8, wherein the updated second score;
second entity comprises a section of code injected by the first in response to evaluating the first entity, when the first
entity. entity satisfies the evaluation criterion, update the first
13. The computer-readable medium of claim 8, wherein score; and
updating the second score comprises changing the second 35 in response, determine whether the first executable entity is
score by an amount determined according to wS, wherein S malicious according to the updated first score.
is the first score, and wherein w is a numerical weight. ck ck ck ck ck