diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 0e64ad78625..8c058440ad7 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -310,7 +310,7 @@ jobs:
uses: ./.github/workflows/build-windows.yml
with:
platform: windows-x64
- msvc-toolset-version: '14.43'
+ msvc-toolset-version: '14.44'
msvc-toolset-architecture: 'x86.x64'
configure-arguments: ${{ github.event.inputs.configure-arguments }}
make-arguments: ${{ github.event.inputs.make-arguments }}
@@ -322,7 +322,7 @@ jobs:
uses: ./.github/workflows/build-windows.yml
with:
platform: windows-aarch64
- msvc-toolset-version: '14.43'
+ msvc-toolset-version: '14.44'
msvc-toolset-architecture: 'arm64'
make-target: 'hotspot'
extra-conf-options: '--openjdk-target=aarch64-unknown-cygwin'
diff --git a/doc/hotspot-style.html b/doc/hotspot-style.html
index d4c06fbd6bd..dafd29d6f54 100644
--- a/doc/hotspot-style.html
+++ b/doc/hotspot-style.html
@@ -77,6 +77,9 @@ HotSpot Coding Style
thread_local
nullptr
<atomic>
+Initializing
+variables with static storage duration
Uniform Initialization
HotSpot Coding Style
Inheriting constructors
Attributes
+noexcept
Additional Permitted
Features
@@ -791,6 +795,33 @@ <atomic>
"conservative" memory ordering, which may differ from (may be stronger
than) sequentially consistent. There are algorithms in HotSpot that are
believed to rely on that ordering.
+Initializing
+variables with static storage duration
+Variables with static storage duration and dynamic
+initialization C++14
+3.6.2). should be avoided, unless an implementation is permitted to
+perform the initialization as a static initialization. The order in
+which dynamic initializations occur is incompletely specified.
+Initialization order problems can be difficult to deal with and lead to
+surprises.
+Variables with static storage duration and non-trivial destructors
+should be avoided. HotSpot doesn't generally try to cleanup on exit, and
+running destructors at exit can lead to problems.
+Some of the approaches used in HotSpot to avoid dynamic
+initialization include:
+
+Use the Deferred<T> class template. Add a call
+to its initialization function at an appropriate place during VM
+initialization. The underlying object is never destroyed.
For objects of class type, use a variable whose value is a
+pointer to the class, initialized to nullptr. Provide an
+initialization function that sets the variable to a dynamically
+allocated object. Add a call to that function at an appropriate place
+during VM initialization. Such objects are usually never
+destroyed.
The use of uniform initialization (n2672),
@@ -1110,6 +1141,58 @@
Attributes
memory_order_consume.
[[deprecated]] - Not relevant in HotSpot code.
+noexcept
+Use of noexcept exception specifications (n3050) are permitted with restrictions
+described below.
+
+- Only the argument-less form of
noexcept exception
+specifications are permitted.
+- Allocation functions that may return
nullptr to
+indicate allocation failure must be declared noexcept.
+- All other uses of
noexcept exception specifications are
+forbidden.
+noexcept expressions are forbidden.- Dynamic exception specifications are forbidden.
+
+HotSpot is built with exceptions disabled, e.g. compile with
+-fno-exceptions (gcc, clang) or no /EH option
+(MSVC++). So why do we need to consider noexcept at all?
+It's because noexcept exception specifications serve two
+distinct purposes.
+The first is to allow the compiler to avoid generating code or data
+in support of exceptions being thrown by a function. But this is
+unnecessary, because exceptions are disabled.
+The second is to allow the compiler and library code to choose
+different algorithms, depending on whether some function may throw
+exceptions. This is only relevant to a certain set of functions.
+
+Some allocation functions (operator new and
+operator new[]) return nullptr to indicate
+allocation failure. If a new expression calls such an
+allocation function, it must check for and handle that possibility.
+Declaring such a function noexcept informs the compiler
+that nullptr is a possible result. If an allocation
+function is not declared noexcept then the compiler may
+elide that checking and handling for a new expression
+calling that function.
Certain Standard Library facilities (notably containers) provide
+different guarantees for some operations (and may choose different
+algorithms to implement those operations), depending on whether certain
+functions (constructors, copy/move operations, swap) are nothrow or not.
+They detect this using type traits that test whether a function is
+declared noexcept. This can have a significant performance
+impact if, for example, copying is chosen over a potentially throwing
+move. But this isn't relevant, since HotSpot forbids the use of most
+Standard Library facilities.
+HotSpot code can assume no exceptions will ever be thrown, even from
+functions not declared noexcept. So HotSpot code doesn't
+ever need to check, either with conditional exception specifications or
+with noexcept expressions.
+Dynamic exception specifications were deprecated in C++11. C++17
+removed all but throw(), with that remaining a deprecated
+equivalent to noexcept.
Additional Permitted
Features
@@ -1198,12 +1281,6 @@ Excluded Features
href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2179.html">n2179)
— HotSpot does not permit the use of exceptions, so this feature isn't
useful.
-Avoid non-local variables with non-constexpr initialization. In
-particular, avoid variables with types requiring non-trivial
-initialization or destruction. Initialization order problems can be
-difficult to deal with and lead to surprises, as can destruction
-ordering. HotSpot doesn't generally try to cleanup on exit, and running
-destructors at exit can also lead to problems.
Avoid most operator overloading, preferring named functions. When
operator overloading is used, ensure the semantics conform to the normal
expected behavior of the operation.
+ // The region has tag `current_tag` and the operation has tag `operation_tag`.
+ // For each state, we decide how much to be added/subtracted from current_tag to operation_tag. Two tables for reserve and commit.
+ // Each pair of in the table means add `x` to current_tag and add `y` to operation_tag. There are 3 pairs in each row for 3 states.
+ // For example, `reserve[1][4,5]` says `-a,a` means:
+ // - we are reserving with operation_tag a region which is already commited with current_tag
+ // - since we are reserving, then `a` will be added to operation_tag. (`y` is `a`)
+ // - since we uncommitting (by reserving) then `a` is to be subtracted from current_tag. (`x` is `-a`).
+ // - amount of uncommitted size is in table `commit[1][4,5]` which is `-a,0` that means subtract `a` from current_tag.
+ // existing state
+ SingleDiff::delta reserve[4][3*2] = {// Rl Rs C
+ {0,0, -a,0, -a,0 }, // op == Release
+ {0,a, -a,a, -a,a }, // op == Reserve
+ {0,a, -a,a, -a,a }, // op == Commit
+ {0,0, 0,0, 0,0 } // op == Uncommit
+ };
+ SingleDiff::delta commit[4][3*2] = {// Rl Rs C
+ {0,0, 0,0, -a,0 }, // op == Release
+ {0,0, 0,0, -a,0 }, // op == Reserve
+ {0,a, 0,a, -a,a }, // op == Commit
+ {0,0, 0,0, -a,0 } // op == Uncommit
+ };
+ SingleDiff& from_rescom = diff.tag[NMTUtil::tag_to_index(current_tag)];
+ SingleDiff& to_rescom = diff.tag[NMTUtil::tag_to_index(operation_tag)];
+ int st = state_to_index(ex);
+ from_rescom.reserve += reserve[op][st * 2 ];
+ to_rescom.reserve += reserve[op][st * 2 + 1];
+ from_rescom.commit += commit[op][st * 2 ];
+ to_rescom.commit += commit[op][st * 2 + 1];
+
+}
+// update the region state between n1 and n2. Since n1 and n2 are pointers, any update of them will be visible from tree.
+// If n1 is noop, it can be removed because its left region (n1->val().in) is already decided and its right state (n1->val().out) is decided here.
+// The state of right of n2 (n2->val().out) cannot be decided here yet.
+void VMATree::update_region(TreapNode* n1, TreapNode* n2, const RequestInfo& req, SummaryDiff& diff) {
+ assert(n1 != nullptr,"sanity");
+ assert(n2 != nullptr,"sanity");
+ //.........n1......n2......
+ // ^------^
+ // |
+ IntervalState exSt = n1->val().out; // existing state info
+
+
+ StateType existing_state = exSt.type();
+ MemTag existing_tag = exSt.mem_tag();
+ SIndex existing_reserve_callstack = exSt.reserved_stack();
+ SIndex existing_commit_callstack = exSt.committed_stack();
+
+ StateType new_state = get_new_state(existing_state, req);
+ MemTag new_tag = get_new_tag(n1->val().out.mem_tag(), req);
+ SIndex new_reserve_callstack = get_new_reserve_callstack(existing_reserve_callstack, existing_state, req);
+ SIndex new_commit_callstack = get_new_commit_callstack(existing_commit_callstack, existing_state, req);
+
+ // n1........n2
+ // out-->
+ n1->val().out.set_tag(new_tag);
+ n1->val().out.set_type(new_state);
+ n1->val().out.set_reserve_stack(new_reserve_callstack);
+ n1->val().out.set_commit_stack(new_commit_callstack);
+
+ // n1........n2
+ // <--in
+ n2->val().in.set_tag(new_tag);
+ n2->val().in.set_type(new_state);
+ n2->val().in.set_reserve_stack(new_reserve_callstack);
+ n2->val().in.set_commit_stack(new_commit_callstack);
+
+ SingleDiff::delta region_size = n2->key() - n1->key();
+ compute_summary_diff(region_size, existing_tag, existing_state, req, new_tag, diff);
+}
+
+
+VMATree::SummaryDiff VMATree::register_mapping(position _A, position _B, StateType state,
const RegionData& metadata, bool use_tag_inplace) {
- assert(!use_tag_inplace || metadata.mem_tag == mtNone,
- "If using use_tag_inplace, then the supplied tag should be mtNone, was instead: %s", NMTUtil::tag_to_name(metadata.mem_tag));
- if (A == B) {
- // A 0-sized mapping isn't worth recording.
+
+ if (_A == _B) {
return SummaryDiff();
}
-
+ assert(_A < _B, "should be");
+ SummaryDiff diff;
+ RequestInfo req{_A, _B, state, metadata.mem_tag, metadata.stack_idx, use_tag_inplace};
IntervalChange stA{
IntervalState{StateType::Released, empty_regiondata},
IntervalState{ state, metadata}
@@ -51,176 +259,401 @@ VMATree::SummaryDiff VMATree::register_mapping(position A, position B, StateType
IntervalState{ state, metadata},
IntervalState{StateType::Released, empty_regiondata}
};
+ stA.out.set_commit_stack(NativeCallStackStorage::invalid);
+ stB.in.set_commit_stack(NativeCallStackStorage::invalid);
+ VMATreap::Range rA = _tree.find_enclosing_range(_A);
+ VMATreap::Range rB = _tree.find_enclosing_range(_B);
- // First handle A.
- // Find closest node that is LEQ A
- bool LEQ_A_found = false;
- AddressState LEQ_A;
- TreapNode* leqA_n = _tree.closest_leq(A);
- if (leqA_n == nullptr) {
- assert(!use_tag_inplace, "Cannot use the tag inplace if no pre-existing tag exists. From: " PTR_FORMAT " To: " PTR_FORMAT, A, B);
- if (use_tag_inplace) {
- log_debug(nmt)("Cannot use the tag inplace if no pre-existing tag exists. From: " PTR_FORMAT " To: " PTR_FORMAT, A, B);
- }
- // No match. We add the A node directly, unless it would have no effect.
- if (!stA.is_noop()) {
- _tree.upsert(A, stA);
+ // nodes: .....X.......Y...Z......W........U
+ // request: A------------------B
+ // X,Y = enclosing_nodes(A)
+ // W,U = enclosing_nodes(B)
+ // The cases are whether or not X and Y exists and X == A. (A == Y doesn't happen since it is searched by 'lt' predicate)
+ // The cases are whether or not W and U exists and W == B. (B == U doesn't happen since it is searched by 'lt' predicate)
+
+ // We update regions in 3 sections: 1) X..A..Y, 2) Y....W, 3) W..B..U
+ // Y: is the closest node greater than A, but less than B
+ // W: is the closest node less than B, but greater than A
+ // The regions in [Y,W) are updated in a loop. We update X..A..Y before the loop and W..B..U after the loop.
+ // The table below summarizes the overlap cases. The overlapping case depends on whether X, Y, W and U exist or not,
+ // and if they exist whether they are the same or not.
+ // In the notations here, when there is not dot ('.') between two nodes it meaans that they are the same. For example,
+ // ...XA....Y.... means X == A.
+
+
+ // row 0: .........A..................B.....
+ // row 1: .........A...YW.............B..... // it is impossible, since it means only one node exists in the tree.
+ // row 2: .........A...Y..........W...B.....
+ // row 3: .........A...Y.............WB.....
+
+ // row 4: .....X...A..................B.....
+ // row 5: .....X...A...YW.............B.....
+ // row 6: .....X...A...Y..........W...B.....
+ // row 7: .....X...A...Y.............WB.....
+
+ // row 8: ........XA..................B.....
+ // row 9: ........XA...YW.............B.....
+ // row 10: ........XA...Y..........W...B.....
+ // row 11: ........XA...Y.............WB.....
+
+ // row 12: .........A..................B....U
+ // row 13: .........A...YW.............B....U
+ // row 14: .........A...Y..........W...B....U
+ // row 15: .........A...Y.............WB....U
+
+ // row 16: .....X...A..................B....U
+ // row 17: .....X...A...YW.............B....U
+ // row 18: .....X...A...Y..........W...B....U
+ // row 19: .....X...A...Y.............WB....U
+
+ // row 20: ........XA..................B....U
+ // row 21: ........XA...YW.............B....U
+ // row 22: ........XA...Y..........W...B....U
+ // row 23: ........XA...Y.............WB....U
+
+
+ // We intentionally did not summarize/compress the cases to keep them as separate.
+ // This expanded way of describing the cases helps us to understand/analyze/verify/debug/maintain
+ // the corresponding code more easily.
+ // Mapping of table to row, row to switch-case should be consistent. If one changes, the others have
+ // to be updated accordingly. The sequence of dependecies is: table -> row no -> switch(row)-case -> code.
+ // Meaning that whenever any of one item in this sequence is changed, the rest of the consequent items to
+ // be checked/changed.
+
+ TreapNode* X = rA.start;
+ TreapNode* Y = rA.end;
+ TreapNode* W = rB.start;
+ TreapNode* U = rB.end;
+ TreapNode nA{_A, stA, 0}; // the node that represents A
+ TreapNode nB{_B, stB, 0}; // the node that represents B
+ TreapNode* A = &nA;
+ TreapNode* B = &nB;
+ auto upsert_if= [&](TreapNode* node) {
+ if (!node->val().is_noop()) {
+ _tree.upsert(node->key(), node->val());
}
- } else {
- LEQ_A_found = true;
- LEQ_A = AddressState{leqA_n->key(), leqA_n->val()};
- StateType leqA_state = leqA_n->val().out.type();
- StateType new_state = stA.out.type();
- // If we specify use_tag_inplace then the new region takes over the current tag instead of the tag in metadata.
- // This is important because the VirtualMemoryTracker API doesn't require supplying the tag for some operations.
- if (use_tag_inplace) {
- assert(leqA_n->val().out.type() != StateType::Released, "Should not use inplace the tag of a released region");
- MemTag tag = leqA_n->val().out.mem_tag();
- stA.out.set_tag(tag);
- stB.in.set_tag(tag);
+ };
+ // update region between n1 and n2
+ auto update = [&](TreapNode* n1, TreapNode* n2) {
+ update_region(n1, n2, req, diff);
+ };
+ auto remove_if = [&](TreapNode* node) -> bool{
+ if (node->val().is_noop()) {
+ _tree.remove(node->key());
+ return true;
}
-
- // Unless we know better, let B's outgoing state be the outgoing state of the node at or preceding A.
- // Consider the case where the found node is the start of a region enclosing [A,B)
- stB.out = out_state(leqA_n);
-
- // Direct address match.
- if (leqA_n->key() == A) {
- // Take over in state from old address.
- stA.in = in_state(leqA_n);
-
- // We may now be able to merge two regions:
- // If the node's old state matches the new, it becomes a noop. That happens, for example,
- // when expanding a committed area: commit [x1, A); ... commit [A, x3)
- // and the result should be a larger area, [x1, x3). In that case, the middle node (A and le_n)
- // is not needed anymore. So we just remove the old node.
- stB.in = stA.out;
- if (stA.is_noop()) {
- // invalidates leqA_n
- _tree.remove(leqA_n->key());
- } else {
- // If the state is not matching then we have different operations, such as:
- // reserve [x1, A); ... commit [A, x2); or
- // reserve [x1, A), mem_tag1; ... reserve [A, x2), mem_tag2; or
- // reserve [A, x1), mem_tag1; ... reserve [A, x2), mem_tag2;
- // then we re-use the existing out node, overwriting its old metadata.
- leqA_n->val() = stA;
- }
- } else {
- // The address must be smaller.
- assert(A > leqA_n->key(), "must be");
-
- // We add a new node, but only if there would be a state change. If there would not be a
- // state change, we just omit the node.
- // That happens, for example, when reserving within an already reserved region with identical metadata.
- stA.in = out_state(leqA_n); // .. and the region's prior state is the incoming state
- if (stA.is_noop()) {
- // Nothing to do.
- } else {
- // Add new node.
- _tree.upsert(A, stA);
+ return false;
+ };
+ GrowableArrayCHeap to_be_removed;
+ // update regions in range A to B
+ auto update_loop = [&]() {
+ TreapNode* prev = nullptr;
+ _tree.visit_range_in_order(_A + 1, _B + 1, [&](TreapNode* curr) {
+ if (prev != nullptr) {
+ update_region(prev, curr, req, diff);
+ // during visit, structure of the tree should not be changed
+ // keep the keys to be removed, and remove them later
+ if (prev->val().is_noop()) {
+ to_be_removed.push(prev->key());
+ }
}
- }
- }
+ prev = curr;
+ return true;
+ });
+ };
+ // update region of [A,T)
+ auto update_A = [&](TreapNode* T) {
+ A->val().out = A->val().in;
+ update(A, T);
+ };
+ bool X_exists = X != nullptr;
+ bool Y_exists = Y != nullptr && Y->key() <= _B;
+ bool W_exists = W != nullptr && W->key() > _A;
+ bool U_exists = U != nullptr;
+ bool X_eq_A = X_exists && X->key() == _A;
+ bool W_eq_B = W_exists && W->key() == _B;
+ bool Y_eq_W = Y_exists && W_exists && W->key() == Y->key();
+ int row = -1;
+#ifdef ASSERT
+ auto print_case = [&]() {
+ log_trace(vmatree)(" req: %4d---%4d", (int)_A, (int)_B);
+ log_trace(vmatree)(" row: %2d", row);
+ log_trace(vmatree)(" X: %4ld", X_exists ? (long)X->key() : -1);
+ log_trace(vmatree)(" Y: %4ld", Y_exists ? (long)Y->key() : -1);
+ log_trace(vmatree)(" W: %4ld", W_exists ? (long)W->key() : -1);
+ log_trace(vmatree)(" U: %4ld", U_exists ? (long)U->key() : -1);
+ };
+#endif
+ // Order of the nodes if they exist are as: X <= A < Y <= W <= B < U
+ // A---------------------------B
+ // X Y YW WB U
+ // XA Y YW WB U
+ if (!X_exists && !Y_exists && !U_exists) { row = 0; }
+ if (!X_exists && Y_exists && Y_eq_W && !W_eq_B && !U_exists) { row = 1; }
+ if (!X_exists && Y_exists && !Y_eq_W && !W_eq_B && !U_exists) { row = 2; }
+ if (!X_exists && Y_exists && W_eq_B && !U_exists) { row = 3; }
- // Now we handle B.
- // We first search all nodes that are (A, B]. All of these nodes
- // need to be deleted and summary accounted for. The last node before B determines B's outgoing state.
- // If there is no node between A and B, its A's incoming state.
- GrowableArrayCHeap to_be_deleted_inbetween_a_b;
- bool B_needs_insert = true;
-
- // Find all nodes between (A, B] and record their addresses and values. Also update B's
- // outgoing state.
- _tree.visit_range_in_order(A + 1, B + 1, [&](TreapNode* head) {
- int cmp_B = PositionComparator::cmp(head->key(), B);
- stB.out = out_state(head);
- if (cmp_B < 0) {
- // Record all nodes preceding B.
- to_be_deleted_inbetween_a_b.push({head->key(), head->val()});
- } else if (cmp_B == 0) {
- // Re-purpose B node, unless it would result in a noop node, in
- // which case record old node at B for deletion and summary accounting.
- if (stB.is_noop()) {
- to_be_deleted_inbetween_a_b.push(AddressState{B, head->val()});
- } else {
- head->val() = stB;
- }
- B_needs_insert = false;
- }
- });
+ if ( X_exists && !Y_exists && !U_exists) { row = 4; }
+ if ( X_exists && Y_exists && Y_eq_W && !W_eq_B && !U_exists) { row = 5; }
+ if ( X_exists && Y_exists && !Y_eq_W && !W_eq_B && !U_exists) { row = 6; }
+ if ( X_exists && Y_exists && W_eq_B && !U_exists) { row = 7; }
- // Insert B node if needed
- if (B_needs_insert && // Was not already inserted
- !stB.is_noop()) // The operation is differing
- {
- _tree.upsert(B, stB);
- }
+ if ( X_eq_A && !Y_exists && !U_exists) { row = 8; }
+ if ( X_eq_A && Y_exists && Y_eq_W && !W_eq_B && !U_exists) { row = 9; }
+ if ( X_eq_A && Y_exists && !Y_eq_W && !W_eq_B && !U_exists) { row = 10; }
+ if ( X_eq_A && Y_exists && W_eq_B && !U_exists) { row = 11; }
- // We now need to:
- // a) Delete all nodes between (A, B]. Including B in the case of a noop.
- // b) Perform summary accounting
- SummaryDiff diff;
+ if (!X_exists && !Y_exists && U_exists) { row = 12; }
+ if (!X_exists && Y_exists && Y_eq_W && !W_eq_B && U_exists) { row = 13; }
+ if (!X_exists && Y_exists && !Y_eq_W && !W_eq_B && U_exists) { row = 14; }
+ if (!X_exists && Y_exists && W_eq_B && U_exists) { row = 15; }
- if (to_be_deleted_inbetween_a_b.length() == 0 && LEQ_A_found) {
- // We must have smashed a hole in an existing region (or replaced it entirely).
- // LEQ_A < A < B <= C
- SingleDiff& rescom = diff.tag[NMTUtil::tag_to_index(LEQ_A.out().mem_tag())];
- if (LEQ_A.out().type() == StateType::Reserved) {
- rescom.reserve -= B - A;
- } else if (LEQ_A.out().type() == StateType::Committed) {
- rescom.commit -= B - A;
- rescom.reserve -= B - A;
- }
- }
+ if ( X_exists && !Y_exists && U_exists) { row = 16; }
+ if ( X_exists && Y_exists && Y_eq_W && !W_eq_B && U_exists) { row = 17; }
+ if ( X_exists && Y_exists && !Y_eq_W && !W_eq_B && U_exists) { row = 18; }
+ if ( X_exists && Y_exists && W_eq_B && U_exists) { row = 19; }
- // Track the previous node.
- AddressState prev{A, stA};
- for (int i = 0; i < to_be_deleted_inbetween_a_b.length(); i++) {
- const AddressState delete_me = to_be_deleted_inbetween_a_b.at(i);
- _tree.remove(delete_me.address);
-
- // Perform summary accounting
- SingleDiff& rescom = diff.tag[NMTUtil::tag_to_index(delete_me.in().mem_tag())];
- if (delete_me.in().type() == StateType::Reserved) {
- rescom.reserve -= delete_me.address - prev.address;
- } else if (delete_me.in().type() == StateType::Committed) {
- rescom.commit -= delete_me.address - prev.address;
- rescom.reserve -= delete_me.address - prev.address;
- }
- prev = delete_me;
- }
+ if ( X_eq_A && !Y_exists && U_exists) { row = 20; }
+ if ( X_eq_A && Y_exists && Y_eq_W && !W_eq_B && U_exists) { row = 21; }
+ if ( X_eq_A && Y_exists && !Y_eq_W && !W_eq_B && U_exists) { row = 22; }
+ if ( X_eq_A && Y_exists && W_eq_B && U_exists) { row = 23; }
- if (prev.address != A && prev.out().type() != StateType::Released) {
- // The last node wasn't released, so it must be connected to a node outside of (A, B)
- // A - prev - B - (some node >= B)
- // It might be that prev.address == B == (some node >= B), this is fine.
- if (prev.out().type() == StateType::Reserved) {
- SingleDiff& rescom = diff.tag[NMTUtil::tag_to_index(prev.out().mem_tag())];
- rescom.reserve -= B - prev.address;
- } else if (prev.out().type() == StateType::Committed) {
- SingleDiff& rescom = diff.tag[NMTUtil::tag_to_index(prev.out().mem_tag())];
- rescom.commit -= B - prev.address;
- rescom.reserve -= B - prev.address;
+ switch(row) {
+ // row 0: .........A..................B.....
+ case 0: {
+ update_A(B);
+ upsert_if(A);
+ upsert_if(B);
+ break;
+ }
+ // row 1: .........A...YW.............B.....
+ case 1: {
+ ShouldNotReachHere();
+ break;
+ }
+ // row 2: .........A...Y..........W...B.....
+ case 2: {
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ remove_if(Y);
+ update(W, B);
+ remove_if(W);
+ upsert_if(B);
+ break;
+ }
+ // row 3: .........A...Y.............WB.....
+ case 3: {
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ remove_if(W);
+ break;
+ }
+ // row 4: .....X...A..................B.....
+ case 4: {
+ A->val().in = X->val().out;
+ update_A(B);
+ upsert_if(A);
+ upsert_if(B);
+ break;
}
+ // row 5: .....X...A...YW.............B.....
+ case 5: {
+ A->val().in = X->val().out;
+ update_A(Y);
+ upsert_if(A);
+ update(Y, B);
+ remove_if(Y);
+ upsert_if(B);
+ break;
+ }
+ // row 6: .....X...A...Y..........W...B.....
+ case 6: {
+ A->val().in = X->val().out;
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ update(W, B);
+ remove_if(W);
+ upsert_if(B);
+ break;
+ }
+ // row 7: .....X...A...Y.............WB.....
+ case 7: {
+ A->val().in = X->val().out;
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ remove_if(W);
+ break;
+ }
+ // row 8: ........XA..................B.....
+ case 8: {
+ update(X, B);
+ remove_if(X);
+ upsert_if(B);
+ break;
+ }
+ // row 9: ........XA...YW.............B.....
+ case 9: {
+ update(X, Y);
+ remove_if(X);
+ update(W, B);
+ remove_if(W);
+ upsert_if(B);
+ break;
+ }
+ // row 10: ........XA...Y..........W...B.....
+ case 10: {
+ update(X, Y);
+ remove_if(X);
+ update_loop();
+ update(W, B);
+ remove_if(W);
+ upsert_if(B);
+ break;
+ }
+ // row 11: ........XA...Y.............WB.....
+ case 11: {
+ update(X, Y);
+ remove_if(X);
+ update_loop();
+ remove_if(W);
+ break;
+ }
+ // row 12: .........A..................B....U
+ case 12: {
+ update_A(B);
+ upsert_if(A);
+ upsert_if(B);
+ break;
+ }
+ // row 13: .........A...YW.............B....U
+ case 13: {
+ update_A(Y);
+ upsert_if(A);
+ update(W, B);
+ remove_if(W);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 14: .........A...Y..........W...B....U
+ case 14: {
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ update(W, B);
+ remove_if(W);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 15: .........A...Y.............WB....U
+ case 15: {
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ remove_if(W);
+ break;
+ }
+ // row 16: .....X...A..................B....U
+ case 16: {
+ A->val().in = X->val().out;
+ update_A(B);
+ upsert_if(A);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 17: .....X...A...YW.............B....U
+ case 17: {
+ A->val().in = X->val().out;
+ update_A(Y);
+ upsert_if(A);
+ update(W, B);
+ remove_if(W);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 18: .....X...A...Y..........W...B....U
+ case 18: {
+ A->val().in = X->val().out;
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ update(W, B);
+ remove_if(W);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 19: .....X...A...Y.............WB....U
+ case 19: {
+ A->val().in = X->val().out;
+ update_A(Y);
+ upsert_if(A);
+ update_loop();
+ remove_if(W);
+ break;
+ }
+ // row 20: ........XA..................B....U
+ case 20: {
+ update(X, B);
+ remove_if(X);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 21: ........XA...YW.............B....U
+ case 21: {
+ update(X, Y);
+ remove_if(X);
+ update(W, B);
+ remove_if(W);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 22: ........XA...Y..........W...B....U
+ case 22: {
+ update(X, Y);
+ remove_if(X);
+ update_loop();
+ update(W, B);
+ remove_if(W);
+ B->val().out = U->val().in;
+ upsert_if(B);
+ break;
+ }
+ // row 23: ........XA...Y.............WB....U
+ case 23: {
+ update(X, Y);
+ remove_if(X);
+ update_loop();
+ remove_if(W);
+ break;
+ }
+ default:
+ ShouldNotReachHere();
}
- // Finally, we can register the new region [A, B)'s summary data.
- SingleDiff& rescom = diff.tag[NMTUtil::tag_to_index(stA.out.mem_tag())];
- if (state == StateType::Reserved) {
- rescom.reserve += B - A;
- } else if (state == StateType::Committed) {
- rescom.commit += B - A;
- rescom.reserve += B - A;
+ // Remove the 'noop' nodes that found inside the loop
+ while(to_be_removed.length() != 0) {
+ _tree.remove(to_be_removed.pop());
}
+
return diff;
}
#ifdef ASSERT
void VMATree::print_on(outputStream* out) {
visit_in_order([&](TreapNode* current) {
- out->print("%zu (%s) - %s - ", current->key(), NMTUtil::tag_to_name(out_state(current).mem_tag()),
- statetype_to_string(out_state(current).type()));
+ out->print("%zu (%s) - %s [%d, %d]-> ", current->key(), NMTUtil::tag_to_name(out_state(current).mem_tag()),
+ statetype_to_string(out_state(current).type()), current->val().out.reserved_stack(), current->val().out.committed_stack());
+ return true;
});
out->cr();
}
@@ -268,7 +701,7 @@ VMATree::SummaryDiff VMATree::set_tag(const position start, const size size, con
SummaryDiff diff;
// Ignore any released ranges, these must be mtNone and have no stack
if (type != StateType::Released) {
- RegionData new_data = RegionData(out.stack(), tag);
+ RegionData new_data = RegionData(out.reserved_stack(), tag);
SummaryDiff result = register_mapping(from, end, type, new_data);
diff.add(result);
}
@@ -289,7 +722,7 @@ VMATree::SummaryDiff VMATree::set_tag(const position start, const size size, con
StateType type = out.type();
if (type != StateType::Released) {
- RegionData new_data = RegionData(out.stack(), tag);
+ RegionData new_data = RegionData(out.reserved_stack(), tag);
SummaryDiff result = register_mapping(from, end, type, new_data);
diff.add(result);
}
diff --git a/src/hotspot/share/nmt/vmatree.hpp b/src/hotspot/share/nmt/vmatree.hpp
index 0c639e929b7..7f52b406309 100644
--- a/src/hotspot/share/nmt/vmatree.hpp
+++ b/src/hotspot/share/nmt/vmatree.hpp
@@ -26,6 +26,7 @@
#ifndef SHARE_NMT_VMATREE_HPP
#define SHARE_NMT_VMATREE_HPP
+#include "nmt/memTag.hpp"
#include "nmt/memTag.hpp"
#include "nmt/nmtNativeCallStackStorage.hpp"
#include "nmt/nmtTreap.hpp"
@@ -40,10 +41,12 @@
// The set of points is stored in a balanced binary tree for efficient querying and updating.
class VMATree {
friend class NMTVMATreeTest;
+ friend class VMTWithVMATreeTest;
// A position in memory.
public:
using position = size_t;
using size = size_t;
+ using SIndex = NativeCallStackStorage::StackIndex;
class PositionComparator {
public:
@@ -55,27 +58,29 @@ class VMATree {
}
};
- enum class StateType : uint8_t { Reserved, Committed, Released, LAST };
+ // Bit fields view: bit 0 for Reserved, bit 1 for Committed.
+ // Setting a region as Committed preserves the Reserved state.
+ enum class StateType : uint8_t { Reserved = 1, Committed = 3, Released = 0, st_number_of_states = 4 };
private:
- static const char* statetype_strings[static_cast(StateType::LAST)];
+ static const char* statetype_strings[static_cast(StateType::st_number_of_states)];
public:
NONCOPYABLE(VMATree);
static const char* statetype_to_string(StateType type) {
- assert(type != StateType::LAST, "must be");
+ assert(type < StateType::st_number_of_states, "must be");
return statetype_strings[static_cast(type)];
}
// Each point has some stack and a tag associated with it.
struct RegionData {
- const NativeCallStackStorage::StackIndex stack_idx;
+ const SIndex stack_idx;
const MemTag mem_tag;
RegionData() : stack_idx(), mem_tag(mtNone) {}
- RegionData(NativeCallStackStorage::StackIndex stack_idx, MemTag mem_tag)
+ RegionData(SIndex stack_idx, MemTag mem_tag)
: stack_idx(stack_idx), mem_tag(mem_tag) {}
static bool equals(const RegionData& a, const RegionData& b) {
@@ -91,15 +96,27 @@ class VMATree {
private:
// Store the type and mem_tag as two bytes
uint8_t type_tag[2];
- NativeCallStackStorage::StackIndex sidx;
+ NativeCallStackStorage::StackIndex _reserved_stack;
+ NativeCallStackStorage::StackIndex _committed_stack;
public:
- IntervalState() : type_tag{0,0}, sidx() {}
+ IntervalState() : type_tag{0,0}, _reserved_stack(NativeCallStackStorage::invalid), _committed_stack(NativeCallStackStorage::invalid) {}
+ IntervalState(const StateType type,
+ const MemTag mt,
+ const NativeCallStackStorage::StackIndex res_stack,
+ const NativeCallStackStorage::StackIndex com_stack) {
+ assert(!(type == StateType::Released) || mt == mtNone, "Released state-type must have memory tag mtNone");
+ type_tag[0] = static_cast(type);
+ type_tag[1] = static_cast(mt);
+ _reserved_stack = res_stack;
+ _committed_stack = com_stack;
+ }
IntervalState(const StateType type, const RegionData data) {
assert(!(type == StateType::Released) || data.mem_tag == mtNone, "Released state-type must have memory tag mtNone");
type_tag[0] = static_cast(type);
type_tag[1] = static_cast(data.mem_tag);
- sidx = data.stack_idx;
+ _reserved_stack = data.stack_idx;
+ _committed_stack = NativeCallStackStorage::invalid;
}
StateType type() const {
@@ -110,16 +127,50 @@ class VMATree {
return static_cast(type_tag[1]);
}
- RegionData regiondata() const {
- return RegionData{sidx, mem_tag()};
+ RegionData reserved_regiondata() const {
+ return RegionData{_reserved_stack, mem_tag()};
+ }
+ RegionData committed_regiondata() const {
+ return RegionData{_committed_stack, mem_tag()};
}
void set_tag(MemTag tag) {
type_tag[1] = static_cast(tag);
}
- NativeCallStackStorage::StackIndex stack() const {
- return sidx;
+ NativeCallStackStorage::StackIndex reserved_stack() const {
+ return _reserved_stack;
+ }
+
+ NativeCallStackStorage::StackIndex committed_stack() const {
+ return _committed_stack;
+ }
+
+ void set_reserve_stack(NativeCallStackStorage::StackIndex idx) {
+ _reserved_stack = idx;
+ }
+
+ void set_commit_stack(NativeCallStackStorage::StackIndex idx) {
+ _committed_stack = idx;
+ }
+
+ bool has_reserved_stack() {
+ return _reserved_stack != NativeCallStackStorage::invalid;
+ }
+
+ bool has_committed_stack() {
+ return _committed_stack != NativeCallStackStorage::invalid;
+ }
+
+ void set_type(StateType t) {
+ type_tag[0] = static_cast(t);
+ }
+
+ bool equals(const IntervalState& other) const {
+ return mem_tag() == other.mem_tag() &&
+ type() == other.type() &&
+ reserved_stack() == other.reserved_stack() &&
+ committed_stack() == other.committed_stack();
}
};
@@ -130,8 +181,14 @@ class VMATree {
IntervalState out;
bool is_noop() {
+ if (in.type() == StateType::Released &&
+ in.type() == out.type() &&
+ in.mem_tag() == out.mem_tag()) {
+ return true;
+ }
return in.type() == out.type() &&
- RegionData::equals(in.regiondata(), out.regiondata());
+ RegionData::equals(in.reserved_regiondata(), out.reserved_regiondata()) &&
+ RegionData::equals(in.committed_regiondata(), out.committed_regiondata());
}
};
@@ -193,8 +250,44 @@ class VMATree {
#endif
};
+ enum Operation {Release, Reserve, Commit, Uncommit};
+ struct RequestInfo {
+ position A, B;
+ StateType _op;
+ MemTag tag;
+ SIndex callstack;
+ bool use_tag_inplace;
+ Operation op() const {
+ return
+ _op == StateType::Reserved && !use_tag_inplace ? Operation::Reserve :
+ _op == StateType::Committed ? Operation::Commit :
+ _op == StateType::Reserved && use_tag_inplace ? Operation::Uncommit :
+ Operation::Release;
+ }
+
+ int op_to_index() const {
+ return
+ _op == StateType::Reserved && !use_tag_inplace ? 1 :
+ _op == StateType::Committed ? 2 :
+ _op == StateType::Reserved && use_tag_inplace ? 3 :
+ 0;
+ }
+ };
+
private:
SummaryDiff register_mapping(position A, position B, StateType state, const RegionData& metadata, bool use_tag_inplace = false);
+ StateType get_new_state(const StateType existinting_state, const RequestInfo& req) const;
+ MemTag get_new_tag(const MemTag existinting_tag, const RequestInfo& req) const;
+ SIndex get_new_reserve_callstack(const SIndex existinting_stack, const StateType ex, const RequestInfo& req) const;
+ SIndex get_new_commit_callstack(const SIndex existinting_stack, const StateType ex, const RequestInfo& req) const;
+ void compute_summary_diff(const SingleDiff::delta region_size, const MemTag t1, const StateType& ex, const RequestInfo& req, const MemTag new_tag, SummaryDiff& diff) const;
+ void update_region(TreapNode* n1, TreapNode* n2, const RequestInfo& req, SummaryDiff& diff);
+ int state_to_index(const StateType st) const {
+ return
+ st == StateType::Released ? 0 :
+ st == StateType::Reserved ? 1 :
+ st == StateType::Committed ? 2 : -1;
+ }
public:
SummaryDiff reserve_mapping(position from, size size, const RegionData& metadata) {
@@ -215,12 +308,8 @@ class VMATree {
return register_mapping(from, from + size, StateType::Reserved, metadata, true);
}
- SummaryDiff release_mapping(position from, size size) {
- return register_mapping(from, from + size, StateType::Released, VMATree::empty_regiondata);
- }
-
- VMATreap& tree() {
- return _tree;
+ SummaryDiff release_mapping(position from, position sz) {
+ return register_mapping(from, from + sz, StateType::Released, VMATree::empty_regiondata);
}
public:
@@ -232,7 +321,10 @@ class VMATree {
#ifdef ASSERT
void print_on(outputStream* out);
#endif
-
+ template
+ void visit_range_in_order(const position& from, const position& to, F f) {
+ _tree.visit_range_in_order(from, to, f);
+ }
+ VMATreap& tree() { return _tree; }
};
-
#endif
diff --git a/src/hotspot/share/oops/compressedKlass.hpp b/src/hotspot/share/oops/compressedKlass.hpp
index 8666d4409e8..a8d79ab838e 100644
--- a/src/hotspot/share/oops/compressedKlass.hpp
+++ b/src/hotspot/share/oops/compressedKlass.hpp
@@ -214,7 +214,7 @@ class CompressedKlassPointers : public AllStatic {
// Can only be used after initialization
static address base() { check_init(_base); return _base; }
- static address base_addr() { return (address)&_base; }
+ static address base_addr() { return (address)&_base; }
static int shift() { check_init(_shift); return _shift; }
static address klass_range_start() { return _klass_range_start; }
diff --git a/src/hotspot/share/oops/constantPool.cpp b/src/hotspot/share/oops/constantPool.cpp
index 5b0ee298ddc..3223c56628e 100644
--- a/src/hotspot/share/oops/constantPool.cpp
+++ b/src/hotspot/share/oops/constantPool.cpp
@@ -1937,18 +1937,20 @@ int ConstantPool::find_matching_entry(int pattern_i,
// Compare this constant pool's bootstrap specifier at idx1 to the constant pool
// cp2's bootstrap specifier at idx2.
bool ConstantPool::compare_operand_to(int idx1, const constantPoolHandle& cp2, int idx2) {
- int k1 = operand_bootstrap_method_ref_index_at(idx1);
- int k2 = cp2->operand_bootstrap_method_ref_index_at(idx2);
+ BSMAttributeEntry* e1 = bsm_attribute_entry(idx1);
+ BSMAttributeEntry* e2 = cp2->bsm_attribute_entry(idx2);
+ int k1 = e1->bootstrap_method_index();
+ int k2 = e2->bootstrap_method_index();
bool match = compare_entry_to(k1, cp2, k2);
if (!match) {
return false;
}
- int argc = operand_argument_count_at(idx1);
- if (argc == cp2->operand_argument_count_at(idx2)) {
+ int argc = e1->argument_count();
+ if (argc == e2->argument_count()) {
for (int j = 0; j < argc; j++) {
- k1 = operand_argument_index_at(idx1, j);
- k2 = cp2->operand_argument_index_at(idx2, j);
+ k1 = e1->argument_index(j);
+ k2 = e2->argument_index(j);
match = compare_entry_to(k1, cp2, k2);
if (!match) {
return false;
diff --git a/src/hotspot/share/oops/constantPool.hpp b/src/hotspot/share/oops/constantPool.hpp
index cc9491d7935..be4a7a474d4 100644
--- a/src/hotspot/share/oops/constantPool.hpp
+++ b/src/hotspot/share/oops/constantPool.hpp
@@ -77,6 +77,43 @@ class CPKlassSlot {
}
};
+class BSMAttributeEntry {
+ friend class ConstantPool;
+ u2 _bootstrap_method_index;
+ u2 _argument_count;
+
+ // The argument indexes are stored right after the object, in a contiguous array.
+ // [ bsmi_0 argc_0 arg_00 arg_01 ... arg_0N bsmi_1 argc_1 arg_10 ... arg_1N ... ]
+ // So in order to find the argument array, jump over ourselves.
+ const u2* argument_indexes() const {
+ return reinterpret_cast(this + 1);
+ }
+ u2* argument_indexes() {
+ return reinterpret_cast(this + 1);
+ }
+ // These are overlays on top of the operands array. Do not construct.
+ BSMAttributeEntry() = delete;
+
+public:
+ // Offsets for SA
+ enum {
+ _bsmi_offset = 0,
+ _argc_offset = 1,
+ _argv_offset = 2
+ };
+
+ int bootstrap_method_index() const {
+ return _bootstrap_method_index;
+ }
+ int argument_count() const {
+ return _argument_count;
+ }
+ int argument_index(int n) const {
+ assert(checked_cast(n) < _argument_count, "oob");
+ return argument_indexes()[n];
+ }
+};
+
class ConstantPool : public Metadata {
friend class VMStructs;
friend class JVMCIVMStructs;
@@ -519,10 +556,6 @@ class ConstantPool : public Metadata {
assert(tag_at(cp_index).has_bootstrap(), "Corrupted constant pool");
return extract_low_short_from_int(*int_at_addr(cp_index));
}
- int bootstrap_operand_base(int cp_index) {
- int bsms_attribute_index = bootstrap_methods_attribute_index(cp_index);
- return operand_offset_at(operands(), bsms_attribute_index);
- }
// The first part of the operands array consists of an index into the second part.
// Extract a 32-bit index value from the first part.
static int operand_offset_at(Array* operands, int bsms_attribute_index) {
@@ -560,47 +593,26 @@ class ConstantPool : public Metadata {
else
return operand_offset_at(operands, nextidx);
}
- int bootstrap_operand_limit(int cp_index) {
- int bsms_attribute_index = bootstrap_methods_attribute_index(cp_index);
- return operand_limit_at(operands(), bsms_attribute_index);
- }
#endif //ASSERT
- // Layout of InvokeDynamic and Dynamic bootstrap method specifier
- // data in second part of operands array. This encodes one record in
- // the BootstrapMethods attribute. The whole specifier also includes
- // the name and type information from the main constant pool entry.
- enum {
- _indy_bsm_offset = 0, // CONSTANT_MethodHandle bsm
- _indy_argc_offset = 1, // u2 argc
- _indy_argv_offset = 2 // u2 argv[argc]
- };
-
// These functions are used in RedefineClasses for CP merge
-
int operand_offset_at(int bsms_attribute_index) {
assert(0 <= bsms_attribute_index &&
bsms_attribute_index < operand_array_length(operands()),
"Corrupted CP operands");
return operand_offset_at(operands(), bsms_attribute_index);
}
- u2 operand_bootstrap_method_ref_index_at(int bsms_attribute_index) {
- int offset = operand_offset_at(bsms_attribute_index);
- return operands()->at(offset + _indy_bsm_offset);
- }
- u2 operand_argument_count_at(int bsms_attribute_index) {
- int offset = operand_offset_at(bsms_attribute_index);
- u2 argc = operands()->at(offset + _indy_argc_offset);
- return argc;
- }
- u2 operand_argument_index_at(int bsms_attribute_index, int j) {
+
+ BSMAttributeEntry* bsm_attribute_entry(int bsms_attribute_index) {
int offset = operand_offset_at(bsms_attribute_index);
- return operands()->at(offset + _indy_argv_offset + j);
+ return reinterpret_cast(operands()->adr_at(offset));
}
+
int operand_next_offset_at(int bsms_attribute_index) {
- int offset = operand_offset_at(bsms_attribute_index) + _indy_argv_offset
- + operand_argument_count_at(bsms_attribute_index);
- return offset;
+ BSMAttributeEntry* bsme = bsm_attribute_entry(bsms_attribute_index);
+ u2* argv_start = bsme->argument_indexes();
+ int offset = argv_start - operands()->data();
+ return offset + bsme->argument_count();
}
// Compare a bootstrap specifier data in the operands arrays
bool compare_operand_to(int bsms_attribute_index1, const constantPoolHandle& cp2,
@@ -617,23 +629,19 @@ class ConstantPool : public Metadata {
u2 bootstrap_method_ref_index_at(int cp_index) {
assert(tag_at(cp_index).has_bootstrap(), "Corrupted constant pool");
- int op_base = bootstrap_operand_base(cp_index);
- return operands()->at(op_base + _indy_bsm_offset);
+ int bsmai = bootstrap_methods_attribute_index(cp_index);
+ return bsm_attribute_entry(bsmai)->bootstrap_method_index();
}
u2 bootstrap_argument_count_at(int cp_index) {
assert(tag_at(cp_index).has_bootstrap(), "Corrupted constant pool");
- int op_base = bootstrap_operand_base(cp_index);
- u2 argc = operands()->at(op_base + _indy_argc_offset);
- DEBUG_ONLY(int end_offset = op_base + _indy_argv_offset + argc;
- int next_offset = bootstrap_operand_limit(cp_index));
- assert(end_offset == next_offset, "matched ending");
- return argc;
+ int bsmai = bootstrap_methods_attribute_index(cp_index);
+ return bsm_attribute_entry(bsmai)->argument_count();
}
u2 bootstrap_argument_index_at(int cp_index, int j) {
- int op_base = bootstrap_operand_base(cp_index);
- DEBUG_ONLY(int argc = operands()->at(op_base + _indy_argc_offset));
- assert((uint)j < (uint)argc, "oob");
- return operands()->at(op_base + _indy_argv_offset + j);
+ int bsmai = bootstrap_methods_attribute_index(cp_index);
+ BSMAttributeEntry* bsme = bsm_attribute_entry(bsmai);
+ assert((uint)j < (uint)bsme->argument_count(), "oob");
+ return bsm_attribute_entry(bsmai)->argument_index(j);
}
// The following methods (name/signature/klass_ref_at, klass_ref_at_noresolve,
diff --git a/src/hotspot/share/oops/fieldInfo.cpp b/src/hotspot/share/oops/fieldInfo.cpp
index 300b45277ad..d0825ba6df8 100644
--- a/src/hotspot/share/oops/fieldInfo.cpp
+++ b/src/hotspot/share/oops/fieldInfo.cpp
@@ -22,8 +22,11 @@
*
*/
+#include "memory/resourceArea.hpp"
+#include "cds/cdsConfig.hpp"
#include "oops/fieldInfo.inline.hpp"
#include "runtime/atomic.hpp"
+#include "utilities/packedTable.hpp"
void FieldInfo::print(outputStream* os, ConstantPool* cp) {
os->print_cr("index=%d name_index=%d name=%s signature_index=%d signature=%s offset=%d "
@@ -37,8 +40,10 @@ void FieldInfo::print(outputStream* os, ConstantPool* cp) {
field_flags().as_uint(),
initializer_index(),
generic_signature_index(),
- _field_flags.is_injected() ? lookup_symbol(generic_signature_index())->as_utf8() : cp->symbol_at(generic_signature_index())->as_utf8(),
- contended_group());
+ _field_flags.is_generic() ? (_field_flags.is_injected() ?
+ lookup_symbol(generic_signature_index())->as_utf8() : cp->symbol_at(generic_signature_index())->as_utf8()
+ ) : "",
+ is_contended() ? contended_group() : 0);
}
void FieldInfo::print_from_growable_array(outputStream* os, GrowableArray* array, ConstantPool* cp) {
@@ -62,13 +67,17 @@ Array* FieldInfoStream::create_FieldInfoStream(GrowableArray* fie
StreamSizer s;
StreamFieldSizer sizer(&s);
+ assert(fields->length() == java_fields + injected_fields, "must be");
+
sizer.consumer()->accept_uint(java_fields);
sizer.consumer()->accept_uint(injected_fields);
for (int i = 0; i < fields->length(); i++) {
FieldInfo* fi = fields->adr_at(i);
sizer.map_field_info(*fi);
}
- int storage_size = sizer.consumer()->position() + 1;
+ // Originally there was an extra byte with 0 terminating the reading;
+ // now we check limits instead.
+ int storage_size = sizer.consumer()->position();
Array* const fis = MetadataFactory::new_array(loader_data, storage_size, CHECK_NULL);
using StreamWriter = UNSIGNED5::Writer*, int, ArrayHelper*, int>>;
@@ -79,15 +88,14 @@ Array* FieldInfoStream::create_FieldInfoStream(GrowableArray* fie
writer.consumer()->accept_uint(java_fields);
writer.consumer()->accept_uint(injected_fields);
for (int i = 0; i < fields->length(); i++) {
- FieldInfo* fi = fields->adr_at(i);
- writer.map_field_info(*fi);
+ writer.map_field_info(fields->at(i));
}
#ifdef ASSERT
FieldInfoReader r(fis);
- int jfc = r.next_uint();
+ int jfc, ifc;
+ r.read_field_counts(&jfc, &ifc);
assert(jfc == java_fields, "Must be");
- int ifc = r.next_uint();
assert(ifc == injected_fields, "Must be");
for (int i = 0; i < jfc + ifc; i++) {
FieldInfo fi;
@@ -113,30 +121,221 @@ Array* FieldInfoStream::create_FieldInfoStream(GrowableArray* fie
return fis;
}
+int FieldInfoStream::compare_name_and_sig(const Symbol* n1, const Symbol* s1, const Symbol* n2, const Symbol* s2) {
+ int cmp = n1->fast_compare(n2);
+ return cmp != 0 ? cmp : s1->fast_compare(s2);
+}
+
+
+// We use both name and signature during the comparison; while JLS require unique
+// names for fields, JVMS requires only unique name + signature combination.
+struct field_pos {
+ Symbol* _name;
+ Symbol* _signature;
+ int _index;
+ int _position;
+};
+
+class FieldInfoSupplier: public PackedTableBuilder::Supplier {
+ const field_pos* _positions;
+ size_t _elements;
+
+public:
+ FieldInfoSupplier(const field_pos* positions, size_t elements): _positions(positions), _elements(elements) {}
+
+ bool next(uint32_t* key, uint32_t* value) override {
+ if (_elements == 0) {
+ return false;
+ }
+ *key = _positions->_position;
+ *value = _positions->_index;
+ ++_positions;
+ --_elements;
+ return true;
+ }
+};
+
+Array* FieldInfoStream::create_search_table(ConstantPool* cp, const Array* fis, ClassLoaderData* loader_data, TRAPS) {
+ if (CDSConfig::is_dumping_dynamic_archive()) {
+ // We cannot use search table; in case of dynamic archives it should be sorted by "requested" addresses,
+ // but Symbol* addresses are coming from _constants, which has "buffered" addresses.
+ // For background, see new comments inside allocate_node_impl in symbolTable.cpp
+ return nullptr;
+ }
+
+ FieldInfoReader r(fis);
+ int java_fields;
+ int injected_fields;
+ r.read_field_counts(&java_fields, &injected_fields);
+ assert(java_fields >= 0, "must be");
+ if (java_fields == 0 || fis->length() == 0 || static_cast(java_fields) < BinarySearchThreshold) {
+ return nullptr;
+ }
+
+ ResourceMark rm;
+ field_pos* positions = NEW_RESOURCE_ARRAY(field_pos, java_fields);
+ for (int i = 0; i < java_fields; ++i) {
+ assert(r.has_next(), "number of fields must match");
+
+ positions[i]._position = r.position();
+ FieldInfo fi;
+ r.read_field_info(fi);
+
+ positions[i]._name = fi.name(cp);
+ positions[i]._signature = fi.signature(cp);
+ positions[i]._index = i;
+ }
+ auto compare_pair = [](const void* v1, const void* v2) {
+ const field_pos* p1 = reinterpret_cast(v1);
+ const field_pos* p2 = reinterpret_cast(v2);
+ return compare_name_and_sig(p1->_name, p1->_signature, p2->_name, p2->_signature);
+ };
+ qsort(positions, java_fields, sizeof(field_pos), compare_pair);
+
+ PackedTableBuilder builder(fis->length() - 1, java_fields - 1);
+ Array* table = MetadataFactory::new_array(loader_data, java_fields * builder.element_bytes(), CHECK_NULL);
+ FieldInfoSupplier supplier(positions, java_fields);
+ builder.fill(table->data(), static_cast(table->length()), supplier);
+ return table;
+}
+
GrowableArray* FieldInfoStream::create_FieldInfoArray(const Array* fis, int* java_fields_count, int* injected_fields_count) {
- int length = FieldInfoStream::num_total_fields(fis);
- GrowableArray* array = new GrowableArray(length);
FieldInfoReader r(fis);
- *java_fields_count = r.next_uint();
- *injected_fields_count = r.next_uint();
+ r.read_field_counts(java_fields_count, injected_fields_count);
+ int length = *java_fields_count + *injected_fields_count;
+
+ GrowableArray* array = new GrowableArray(length);
while (r.has_next()) {
FieldInfo fi;
r.read_field_info(fi);
array->append(fi);
}
assert(array->length() == length, "Must be");
- assert(array->length() == *java_fields_count + *injected_fields_count, "Must be");
return array;
}
void FieldInfoStream::print_from_fieldinfo_stream(Array* fis, outputStream* os, ConstantPool* cp) {
- int length = FieldInfoStream::num_total_fields(fis);
FieldInfoReader r(fis);
- int java_field_count = r.next_uint();
- int injected_fields_count = r.next_uint();
+ int java_fields_count;
+ int injected_fields_count;
+ r.read_field_counts(&java_fields_count, &injected_fields_count);
while (r.has_next()) {
FieldInfo fi;
r.read_field_info(fi);
fi.print(os, cp);
}
}
+
+class FieldInfoComparator: public PackedTableLookup::Comparator {
+ const FieldInfoReader* _reader;
+ ConstantPool* _cp;
+ const Symbol* _name;
+ const Symbol* _signature;
+
+public:
+ FieldInfoComparator(const FieldInfoReader* reader, ConstantPool* cp, const Symbol* name, const Symbol* signature):
+ _reader(reader), _cp(cp), _name(name), _signature(signature) {}
+
+ int compare_to(uint32_t position) override {
+ FieldInfoReader r2(*_reader);
+ r2.set_position_and_next_index(position, -1);
+ u2 name_index, sig_index;
+ r2.read_name_and_signature(&name_index, &sig_index);
+ Symbol* mid_name = _cp->symbol_at(name_index);
+ Symbol* mid_sig = _cp->symbol_at(sig_index);
+
+ return FieldInfoStream::compare_name_and_sig(_name, _signature, mid_name, mid_sig);
+ }
+
+#ifdef ASSERT
+ void reset(uint32_t position) override {
+ FieldInfoReader r2(*_reader);
+ r2.set_position_and_next_index(position, -1);
+ u2 name_index, signature_index;
+ r2.read_name_and_signature(&name_index, &signature_index);
+ _name = _cp->symbol_at(name_index);
+ _signature = _cp->symbol_at(signature_index);
+ }
+#endif // ASSERT
+};
+
+#ifdef ASSERT
+void FieldInfoStream::validate_search_table(ConstantPool* cp, const Array* fis, const Array* search_table) {
+ if (search_table == nullptr) {
+ return;
+ }
+ FieldInfoReader reader(fis);
+ int java_fields, injected_fields;
+ reader.read_field_counts(&java_fields, &injected_fields);
+ assert(java_fields > 0, "must be");
+
+ PackedTableLookup lookup(fis->length() - 1, java_fields - 1, search_table);
+ assert(lookup.element_bytes() * java_fields == static_cast(search_table->length()), "size does not match");
+
+ FieldInfoComparator comparator(&reader, cp, nullptr, nullptr);
+ // Check 1: assert that elements have the correct order based on the comparison function
+ lookup.validate_order(comparator);
+
+ // Check 2: Iterate through the original stream (not just search_table) and try if lookup works as expected
+ reader.set_position_and_next_index(0, 0);
+ reader.read_field_counts(&java_fields, &injected_fields);
+ while (reader.has_next()) {
+ int field_start = reader.position();
+ FieldInfo fi;
+ reader.read_field_info(fi);
+ if (fi.field_flags().is_injected()) {
+ // checking only java fields that precede injected ones
+ break;
+ }
+
+ FieldInfoReader r2(fis);
+ int index = r2.search_table_lookup(search_table, fi.name(cp), fi.signature(cp), cp, java_fields);
+ assert(index == static_cast(fi.index()), "wrong index: %d != %u", index, fi.index());
+ assert(index == r2.next_index(), "index should match");
+ assert(field_start == r2.position(), "must find the same position");
+ }
+}
+#endif // ASSERT
+
+void FieldInfoStream::print_search_table(outputStream* st, ConstantPool* cp, const Array* fis, const Array* search_table) {
+ if (search_table == nullptr) {
+ return;
+ }
+ FieldInfoReader reader(fis);
+ int java_fields, injected_fields;
+ reader.read_field_counts(&java_fields, &injected_fields);
+ assert(java_fields > 0, "must be");
+ PackedTableLookup lookup(fis->length() - 1, java_fields - 1, search_table);
+ auto printer = [&] (size_t offset, uint32_t position, uint32_t index) {
+ reader.set_position_and_next_index(position, -1);
+ u2 name_index, sig_index;
+ reader.read_name_and_signature(&name_index, &sig_index);
+ Symbol* name = cp->symbol_at(name_index);
+ Symbol* sig = cp->symbol_at(sig_index);
+ st->print(" [%zu] #%d,#%d = ", offset, name_index, sig_index);
+ name->print_symbol_on(st);
+ st->print(":");
+ sig->print_symbol_on(st);
+ st->print(" @ %p,%p", name, sig);
+ st->cr();
+ };
+
+ lookup.iterate(printer);
+}
+
+int FieldInfoReader::search_table_lookup(const Array* search_table, const Symbol* name, const Symbol* signature, ConstantPool* cp, int java_fields) {
+ assert(java_fields >= 0, "must be");
+ if (java_fields == 0) {
+ return -1;
+ }
+ FieldInfoComparator comp(this, cp, name, signature);
+ PackedTableLookup lookup(_r.limit() - 1, java_fields - 1, search_table);
+ uint32_t position;
+ static_assert(sizeof(uint32_t) == sizeof(_next_index), "field size assert");
+ if (lookup.search(comp, &position, reinterpret_cast(&_next_index))) {
+ _r.set_position(static_cast(position));
+ return _next_index;
+ } else {
+ return -1;
+ }
+}
diff --git a/src/hotspot/share/oops/fieldInfo.hpp b/src/hotspot/share/oops/fieldInfo.hpp
index 8740d539c8f..a98895da9cf 100644
--- a/src/hotspot/share/oops/fieldInfo.hpp
+++ b/src/hotspot/share/oops/fieldInfo.hpp
@@ -222,29 +222,28 @@ class Mapper {
void map_field_info(const FieldInfo& fi);
};
-
// Gadget for decoding and reading the stream of field records.
class FieldInfoReader {
- friend class FieldInfoStream;
- friend class ClassFileParser;
- friend class FieldStreamBase;
- friend class FieldInfo;
-
UNSIGNED5::Reader _r;
int _next_index;
- public:
+public:
FieldInfoReader(const Array* fi);
- private:
- uint32_t next_uint() { return _r.next_uint(); }
+private:
+ inline uint32_t next_uint() { return _r.next_uint(); }
void skip(int n) { int s = _r.try_skip(n); assert(s == n,""); }
public:
- int has_next() { return _r.has_next(); }
- int position() { return _r.position(); }
- int next_index() { return _next_index; }
+ void read_field_counts(int* java_fields, int* injected_fields);
+ int has_next() const { return _r.position() < _r.limit(); }
+ int position() const { return _r.position(); }
+ int next_index() const { return _next_index; }
+ void read_name_and_signature(u2* name_index, u2* signature_index);
void read_field_info(FieldInfo& fi);
+
+ int search_table_lookup(const Array* search_table, const Symbol* name, const Symbol* signature, ConstantPool* cp, int java_fields);
+
// skip a whole field record, both required and optional bits
FieldInfoReader& skip_field_info();
@@ -271,6 +270,11 @@ class FieldInfoStream : AllStatic {
friend class JavaFieldStream;
friend class FieldStreamBase;
friend class ClassFileParser;
+ friend class FieldInfoReader;
+ friend class FieldInfoComparator;
+
+ private:
+ static int compare_name_and_sig(const Symbol* n1, const Symbol* s1, const Symbol* n2, const Symbol* s2);
public:
static int num_java_fields(const Array* fis);
@@ -278,9 +282,14 @@ class FieldInfoStream : AllStatic {
static int num_total_fields(const Array* fis);
static Array* create_FieldInfoStream(GrowableArray* fields, int java_fields, int injected_fields,
- ClassLoaderData* loader_data, TRAPS);
+ ClassLoaderData* loader_data, TRAPS);
+ static Array* create_search_table(ConstantPool* cp, const Array* fis, ClassLoaderData* loader_data, TRAPS);
static GrowableArray* create_FieldInfoArray(const Array* fis, int* java_fields_count, int* injected_fields_count);
static void print_from_fieldinfo_stream(Array* fis, outputStream* os, ConstantPool* cp);
+
+ DEBUG_ONLY(static void validate_search_table(ConstantPool* cp, const Array* fis, const Array* search_table);)
+
+ static void print_search_table(outputStream* st, ConstantPool* cp, const Array* fis, const Array* search_table);
};
class FieldStatus {
diff --git a/src/hotspot/share/oops/fieldInfo.inline.hpp b/src/hotspot/share/oops/fieldInfo.inline.hpp
index d3d4d765081..842393729b2 100644
--- a/src/hotspot/share/oops/fieldInfo.inline.hpp
+++ b/src/hotspot/share/oops/fieldInfo.inline.hpp
@@ -56,16 +56,27 @@ inline Symbol* FieldInfo::lookup_symbol(int symbol_index) const {
inline int FieldInfoStream::num_injected_java_fields(const Array* fis) {
FieldInfoReader fir(fis);
- fir.skip(1);
- return fir.next_uint();
+ int java_fields_count;
+ int injected_fields_count;
+ fir.read_field_counts(&java_fields_count, &injected_fields_count);
+ return injected_fields_count;
}
inline int FieldInfoStream::num_total_fields(const Array* fis) {
FieldInfoReader fir(fis);
- return fir.next_uint() + fir.next_uint();
+ int java_fields_count;
+ int injected_fields_count;
+ fir.read_field_counts(&java_fields_count, &injected_fields_count);
+ return java_fields_count + injected_fields_count;
}
-inline int FieldInfoStream::num_java_fields(const Array* fis) { return FieldInfoReader(fis).next_uint(); }
+inline int FieldInfoStream::num_java_fields(const Array* fis) {
+ FieldInfoReader fir(fis);
+ int java_fields_count;
+ int injected_fields_count;
+ fir.read_field_counts(&java_fields_count, &injected_fields_count);
+ return java_fields_count;
+}
template
inline void Mapper::map_field_info(const FieldInfo& fi) {
@@ -94,13 +105,22 @@ inline void Mapper::map_field_info(const FieldInfo& fi) {
inline FieldInfoReader::FieldInfoReader(const Array* fi)
- : _r(fi->data(), 0),
+ : _r(fi->data(), fi->length()),
_next_index(0) { }
+inline void FieldInfoReader::read_field_counts(int* java_fields, int* injected_fields) {
+ *java_fields = next_uint();
+ *injected_fields = next_uint();
+}
+
+inline void FieldInfoReader::read_name_and_signature(u2* name_index, u2* signature_index) {
+ *name_index = checked_cast(next_uint());
+ *signature_index = checked_cast(next_uint());
+}
+
inline void FieldInfoReader::read_field_info(FieldInfo& fi) {
fi._index = _next_index++;
- fi._name_index = checked_cast(next_uint());
- fi._signature_index = checked_cast(next_uint());
+ read_name_and_signature(&fi._name_index, &fi._signature_index);
fi._offset = next_uint();
fi._access_flags = AccessFlags(checked_cast(next_uint()));
fi._field_flags = FieldInfo::FieldFlags(next_uint());
diff --git a/src/hotspot/share/oops/fieldStreams.hpp b/src/hotspot/share/oops/fieldStreams.hpp
index a1c5d77eeb6..0ae828d73d9 100644
--- a/src/hotspot/share/oops/fieldStreams.hpp
+++ b/src/hotspot/share/oops/fieldStreams.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2011, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2011, 2025, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -56,17 +56,23 @@ class FieldStreamBase : public StackObj {
inline FieldStreamBase(const Array* fieldinfo_stream, ConstantPool* constants, int start, int limit);
- inline FieldStreamBase(Array* fieldinfo_stream, ConstantPool* constants);
+ inline FieldStreamBase(const Array* fieldinfo_stream, ConstantPool* constants);
- private:
+ private:
void initialize() {
- int java_fields_count = _reader.next_uint();
- int injected_fields_count = _reader.next_uint();
- assert( _limit <= java_fields_count + injected_fields_count, "Safety check");
+ int java_fields_count;
+ int injected_fields_count;
+ _reader.read_field_counts(&java_fields_count, &injected_fields_count);
+ if (_limit < _index) {
+ _limit = java_fields_count + injected_fields_count;
+ } else {
+ assert( _limit <= java_fields_count + injected_fields_count, "Safety check");
+ }
if (_limit != 0) {
_reader.read_field_info(_fi_buf);
}
}
+
public:
inline FieldStreamBase(InstanceKlass* klass);
@@ -138,8 +144,11 @@ class FieldStreamBase : public StackObj {
// Iterate over only the Java fields
class JavaFieldStream : public FieldStreamBase {
+ Array* _search_table;
+
public:
- JavaFieldStream(const InstanceKlass* k): FieldStreamBase(k->fieldinfo_stream(), k->constants(), 0, k->java_fields_count()) {}
+ JavaFieldStream(const InstanceKlass* k): FieldStreamBase(k->fieldinfo_stream(), k->constants(), 0, k->java_fields_count()),
+ _search_table(k->fieldinfo_search_table()) {}
u2 name_index() const {
assert(!field()->field_flags().is_injected(), "regular only");
@@ -149,7 +158,6 @@ class JavaFieldStream : public FieldStreamBase {
u2 signature_index() const {
assert(!field()->field_flags().is_injected(), "regular only");
return field()->signature_index();
- return -1;
}
u2 generic_signature_index() const {
@@ -164,6 +172,10 @@ class JavaFieldStream : public FieldStreamBase {
assert(!field()->field_flags().is_injected(), "regular only");
return field()->initializer_index();
}
+
+ // Performs either a linear search or binary search through the stream
+ // looking for a matching name/signature combo
+ bool lookup(const Symbol* name, const Symbol* signature);
};
@@ -176,7 +188,6 @@ class InternalFieldStream : public FieldStreamBase {
class AllFieldStream : public FieldStreamBase {
public:
- AllFieldStream(Array* fieldinfo, ConstantPool* constants): FieldStreamBase(fieldinfo, constants) {}
AllFieldStream(const InstanceKlass* k): FieldStreamBase(k->fieldinfo_stream(), k->constants()) {}
};
diff --git a/src/hotspot/share/oops/fieldStreams.inline.hpp b/src/hotspot/share/oops/fieldStreams.inline.hpp
index 776bcd1671c..51faf88c678 100644
--- a/src/hotspot/share/oops/fieldStreams.inline.hpp
+++ b/src/hotspot/share/oops/fieldStreams.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2019, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2019, 2025, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -33,22 +33,18 @@
FieldStreamBase::FieldStreamBase(const Array* fieldinfo_stream, ConstantPool* constants, int start, int limit) :
_fieldinfo_stream(fieldinfo_stream),
_reader(FieldInfoReader(_fieldinfo_stream)),
- _constants(constantPoolHandle(Thread::current(), constants)), _index(start) {
- _index = start;
- if (limit < start) {
- _limit = FieldInfoStream::num_total_fields(_fieldinfo_stream);
- } else {
- _limit = limit;
- }
+ _constants(constantPoolHandle(Thread::current(), constants)),
+ _index(start),
+ _limit(limit) {
initialize();
}
-FieldStreamBase::FieldStreamBase(Array* fieldinfo_stream, ConstantPool* constants) :
+FieldStreamBase::FieldStreamBase(const Array* fieldinfo_stream, ConstantPool* constants) :
_fieldinfo_stream(fieldinfo_stream),
_reader(FieldInfoReader(_fieldinfo_stream)),
_constants(constantPoolHandle(Thread::current(), constants)),
_index(0),
- _limit(FieldInfoStream::num_total_fields(_fieldinfo_stream)) {
+ _limit(-1) {
initialize();
}
@@ -57,9 +53,28 @@ FieldStreamBase::FieldStreamBase(InstanceKlass* klass) :
_reader(FieldInfoReader(_fieldinfo_stream)),
_constants(constantPoolHandle(Thread::current(), klass->constants())),
_index(0),
- _limit(FieldInfoStream::num_total_fields(_fieldinfo_stream)) {
+ _limit(-1) {
assert(klass == field_holder(), "");
initialize();
}
+inline bool JavaFieldStream::lookup(const Symbol* name, const Symbol* signature) {
+ if (_search_table != nullptr) {
+ int index = _reader.search_table_lookup(_search_table, name, signature, _constants(), _limit);
+ if (index >= 0) {
+ assert(index < _limit, "must be");
+ _index = index;
+ _reader.read_field_info(_fi_buf);
+ return true;
+ }
+ } else {
+ for (; !done(); next()) {
+ if (this->name() == name && this->signature() == signature) {
+ return true;
+ }
+ }
+ }
+ return false;
+}
+
#endif // SHARE_OOPS_FIELDSTREAMS_INLINE_HPP
diff --git a/src/hotspot/share/oops/instanceKlass.cpp b/src/hotspot/share/oops/instanceKlass.cpp
index 32d7943e1e8..f4ab8c31409 100644
--- a/src/hotspot/share/oops/instanceKlass.cpp
+++ b/src/hotspot/share/oops/instanceKlass.cpp
@@ -686,6 +686,11 @@ void InstanceKlass::deallocate_contents(ClassLoaderData* loader_data) {
}
set_fieldinfo_stream(nullptr);
+ if (fieldinfo_search_table() != nullptr && !fieldinfo_search_table()->is_shared()) {
+ MetadataFactory::free_array(loader_data, fieldinfo_search_table());
+ }
+ set_fieldinfo_search_table(nullptr);
+
if (fields_status() != nullptr && !fields_status()->is_shared()) {
MetadataFactory::free_array(loader_data, fields_status());
}
@@ -1786,13 +1791,12 @@ FieldInfo InstanceKlass::field(int index) const {
}
bool InstanceKlass::find_local_field(Symbol* name, Symbol* sig, fieldDescriptor* fd) const {
- for (JavaFieldStream fs(this); !fs.done(); fs.next()) {
- Symbol* f_name = fs.name();
- Symbol* f_sig = fs.signature();
- if (f_name == name && f_sig == sig) {
- fd->reinitialize(const_cast(this), fs.to_FieldInfo());
- return true;
- }
+ JavaFieldStream fs(this);
+ if (fs.lookup(name, sig)) {
+ assert(fs.name() == name, "name must match");
+ assert(fs.signature() == sig, "signature must match");
+ fd->reinitialize(const_cast(this), fs.to_FieldInfo());
+ return true;
}
return false;
}
@@ -2391,13 +2395,26 @@ jmethodID InstanceKlass::update_jmethod_id(jmethodID* jmeths, Method* method, in
return new_id;
}
+// Allocate the jmethodID cache.
+static jmethodID* create_jmethod_id_cache(size_t size) {
+ jmethodID* jmeths = NEW_C_HEAP_ARRAY(jmethodID, size + 1, mtClass);
+ memset(jmeths, 0, (size + 1) * sizeof(jmethodID));
+ // cache size is stored in element[0], other elements offset by one
+ jmeths[0] = (jmethodID)size;
+ return jmeths;
+}
+
+// When reading outside a lock, use this.
+jmethodID* InstanceKlass::methods_jmethod_ids_acquire() const {
+ return Atomic::load_acquire(&_methods_jmethod_ids);
+}
+
+void InstanceKlass::release_set_methods_jmethod_ids(jmethodID* jmeths) {
+ Atomic::release_store(&_methods_jmethod_ids, jmeths);
+}
+
// Lookup or create a jmethodID.
-// This code is called by the VMThread and JavaThreads so the
-// locking has to be done very carefully to avoid deadlocks
-// and/or other cache consistency problems.
-//
-jmethodID InstanceKlass::get_jmethod_id(const methodHandle& method_h) {
- Method* method = method_h();
+jmethodID InstanceKlass::get_jmethod_id(Method* method) {
int idnum = method->method_idnum();
jmethodID* jmeths = methods_jmethod_ids_acquire();
@@ -2418,15 +2435,12 @@ jmethodID InstanceKlass::get_jmethod_id(const methodHandle& method_h) {
if (jmeths == nullptr) {
MutexLocker ml(JmethodIdCreation_lock, Mutex::_no_safepoint_check_flag);
- jmeths = methods_jmethod_ids_acquire();
+ jmeths = _methods_jmethod_ids;
// Still null?
if (jmeths == nullptr) {
size_t size = idnum_allocated_count();
assert(size > (size_t)idnum, "should already have space");
- jmeths = NEW_C_HEAP_ARRAY(jmethodID, size + 1, mtClass);
- memset(jmeths, 0, (size + 1) * sizeof(jmethodID));
- // cache size is stored in element[0], other elements offset by one
- jmeths[0] = (jmethodID)size;
+ jmeths = create_jmethod_id_cache(size);
jmethodID new_id = update_jmethod_id(jmeths, method, idnum);
// publish jmeths
@@ -2456,10 +2470,7 @@ void InstanceKlass::update_methods_jmethod_cache() {
if (old_size < size + 1) {
// Allocate a larger one and copy entries to the new one.
// They've already been updated to point to new methods where applicable (i.e., not obsolete).
- jmethodID* new_cache = NEW_C_HEAP_ARRAY(jmethodID, size + 1, mtClass);
- memset(new_cache, 0, (size + 1) * sizeof(jmethodID));
- // The cache size is stored in element[0]; the other elements are offset by one.
- new_cache[0] = (jmethodID)size;
+ jmethodID* new_cache = create_jmethod_id_cache(size);
for (int i = 1; i <= (int)old_size; i++) {
new_cache[i] = cache[i];
@@ -2470,24 +2481,30 @@ void InstanceKlass::update_methods_jmethod_cache() {
}
}
-// Figure out how many jmethodIDs haven't been allocated, and make
-// sure space for them is pre-allocated. This makes getting all
-// method ids much, much faster with classes with more than 8
+// Make a jmethodID for all methods in this class. This makes getting all method
+// ids much, much faster with classes with more than 8
// methods, and has a *substantial* effect on performance with jvmti
// code that loads all jmethodIDs for all classes.
-void InstanceKlass::ensure_space_for_methodids(int start_offset) {
- int new_jmeths = 0;
+void InstanceKlass::make_methods_jmethod_ids() {
+ MutexLocker ml(JmethodIdCreation_lock, Mutex::_no_safepoint_check_flag);
+ jmethodID* jmeths = _methods_jmethod_ids;
+ if (jmeths == nullptr) {
+ jmeths = create_jmethod_id_cache(idnum_allocated_count());
+ release_set_methods_jmethod_ids(jmeths);
+ }
+
int length = methods()->length();
- for (int index = start_offset; index < length; index++) {
+ for (int index = 0; index < length; index++) {
Method* m = methods()->at(index);
- jmethodID id = m->find_jmethod_id_or_null();
- if (id == nullptr) {
- new_jmeths++;
+ int idnum = m->method_idnum();
+ assert(!m->is_old(), "should not have old methods or I'm confused");
+ jmethodID id = Atomic::load_acquire(&jmeths[idnum + 1]);
+ if (!m->is_overpass() && // skip overpasses
+ id == nullptr) {
+ id = Method::make_jmethod_id(class_loader_data(), m);
+ Atomic::release_store(&jmeths[idnum + 1], id);
}
}
- if (new_jmeths != 0) {
- Method::ensure_jmethod_ids(class_loader_data(), new_jmeths);
- }
}
// Lookup a jmethodID, null if not found. Do no blocking, no allocations, no handles
@@ -2610,6 +2627,7 @@ void InstanceKlass::metaspace_pointers_do(MetaspaceClosure* it) {
}
it->push(&_fieldinfo_stream);
+ it->push(&_fieldinfo_search_table);
// _fields_status might be written into by Rewriter::scan_method() -> fd.set_has_initialized_final_update()
it->push(&_fields_status, MetaspaceClosure::_writable);
@@ -2710,6 +2728,8 @@ void InstanceKlass::remove_unshareable_info() {
DEBUG_ONLY(_shared_class_load_count = 0);
remove_unshareable_flags();
+
+ DEBUG_ONLY(FieldInfoStream::validate_search_table(_constants, _fieldinfo_stream, _fieldinfo_search_table));
}
void InstanceKlass::remove_unshareable_flags() {
@@ -2816,6 +2836,8 @@ void InstanceKlass::restore_unshareable_info(ClassLoaderData* loader_data, Handl
if (DiagnoseSyncOnValueBasedClasses && has_value_based_class_annotation() && !is_value_based()) {
set_is_value_based();
}
+
+ DEBUG_ONLY(FieldInfoStream::validate_search_table(_constants, _fieldinfo_stream, _fieldinfo_search_table));
}
// Check if a class or any of its supertypes has a version older than 50.
@@ -2914,7 +2936,7 @@ void InstanceKlass::release_C_heap_structures(bool release_sub_metadata) {
JNIid::deallocate(jni_ids());
set_jni_ids(nullptr);
- jmethodID* jmeths = methods_jmethod_ids_acquire();
+ jmethodID* jmeths = _methods_jmethod_ids;
if (jmeths != nullptr) {
release_set_methods_jmethod_ids(nullptr);
FreeHeap(jmeths);
@@ -3492,7 +3514,7 @@ void InstanceKlass::add_osr_nmethod(nmethod* n) {
for (int l = CompLevel_limited_profile; l < n->comp_level(); l++) {
nmethod *inv = lookup_osr_nmethod(n->method(), n->osr_entry_bci(), l, true);
if (inv != nullptr && inv->is_in_use()) {
- inv->make_not_entrant(nmethod::ChangeReason::OSR_invalidation_of_lower_level);
+ inv->make_not_entrant(nmethod::InvalidationReason::OSR_INVALIDATION_OF_LOWER_LEVEL);
}
}
}
@@ -3760,6 +3782,11 @@ void InstanceKlass::print_on(outputStream* st) const {
map++;
}
st->cr();
+
+ if (fieldinfo_search_table() != nullptr) {
+ st->print_cr(BULLET"---- field info search table:");
+ FieldInfoStream::print_search_table(st, _constants, _fieldinfo_stream, _fieldinfo_search_table);
+ }
}
void InstanceKlass::print_value_on(outputStream* st) const {
@@ -4261,17 +4288,14 @@ bool InstanceKlass::should_clean_previous_versions_and_reset() {
return ret;
}
-// This nulls out jmethodIDs for all methods in 'klass'
-// It needs to be called explicitly for all previous versions of a class because these may not be cleaned up
-// during class unloading.
-// We can not use the jmethodID cache associated with klass directly because the 'previous' versions
-// do not have the jmethodID cache filled in. Instead, we need to lookup jmethodID for each method and this
-// is expensive - O(n) for one jmethodID lookup. For all contained methods it is O(n^2).
-// The reason for expensive jmethodID lookup for each method is that there is no direct link between method and jmethodID.
-void InstanceKlass::clear_jmethod_ids(InstanceKlass* klass) {
+// This nulls out the jmethodID for all obsolete methods in the previous version of the 'klass'.
+// These obsolete methods only exist in the previous version and we're about to delete the memory for them.
+// The jmethodID for these are deallocated when we unload the class, so this doesn't remove them from the table.
+void InstanceKlass::clear_obsolete_jmethod_ids(InstanceKlass* klass) {
Array* method_refs = klass->methods();
for (int k = 0; k < method_refs->length(); k++) {
Method* method = method_refs->at(k);
+ // Only need to clear obsolete methods.
if (method != nullptr && method->is_obsolete()) {
method->clear_jmethod_id();
}
@@ -4321,7 +4345,7 @@ void InstanceKlass::purge_previous_version_list() {
// Unlink from previous version list.
assert(pv_node->class_loader_data() == loader_data, "wrong loader_data");
InstanceKlass* next = pv_node->previous_versions();
- clear_jmethod_ids(pv_node); // jmethodID maintenance for the unloaded class
+ clear_obsolete_jmethod_ids(pv_node); // jmethodID maintenance for the unloaded class
pv_node->link_previous_versions(nullptr); // point next to null
last->link_previous_versions(next);
// Delete this node directly. Nothing is referring to it and we don't
diff --git a/src/hotspot/share/oops/instanceKlass.hpp b/src/hotspot/share/oops/instanceKlass.hpp
index 078c5b81841..2512d8d869d 100644
--- a/src/hotspot/share/oops/instanceKlass.hpp
+++ b/src/hotspot/share/oops/instanceKlass.hpp
@@ -276,6 +276,7 @@ class InstanceKlass: public Klass {
// Fields information is stored in an UNSIGNED5 encoded stream (see fieldInfo.hpp)
Array* _fieldinfo_stream;
+ Array* _fieldinfo_search_table;
Array* _fields_status;
// embedded Java vtable follows here
@@ -398,6 +399,9 @@ class InstanceKlass: public Klass {
Array* fieldinfo_stream() const { return _fieldinfo_stream; }
void set_fieldinfo_stream(Array* fis) { _fieldinfo_stream = fis; }
+ Array* fieldinfo_search_table() const { return _fieldinfo_search_table; }
+ void set_fieldinfo_search_table(Array* table) { _fieldinfo_search_table = table; }
+
Array* fields_status() const {return _fields_status; }
void set_fields_status(Array* array) { _fields_status = array; }
@@ -777,8 +781,8 @@ class InstanceKlass: public Klass {
u2 method_index);
// jmethodID support
- jmethodID get_jmethod_id(const methodHandle& method_h);
- void ensure_space_for_methodids(int start_offset = 0);
+ jmethodID get_jmethod_id(Method* method);
+ void make_methods_jmethod_ids();
jmethodID jmethod_id_or_null(Method* method);
void update_methods_jmethod_cache();
@@ -1052,10 +1056,10 @@ class InstanceKlass: public Klass {
Atomic::store(&_init_thread, thread);
}
- inline jmethodID* methods_jmethod_ids_acquire() const;
- inline void release_set_methods_jmethod_ids(jmethodID* jmeths);
- // This nulls out jmethodIDs for all methods in 'klass'
- static void clear_jmethod_ids(InstanceKlass* klass);
+ jmethodID* methods_jmethod_ids_acquire() const;
+ void release_set_methods_jmethod_ids(jmethodID* jmeths);
+ // This nulls out obsolete jmethodIDs for all methods in 'klass'.
+ static void clear_obsolete_jmethod_ids(InstanceKlass* klass);
jmethodID update_jmethod_id(jmethodID* jmeths, Method* method, int idnum);
public:
diff --git a/src/hotspot/share/oops/instanceKlass.inline.hpp b/src/hotspot/share/oops/instanceKlass.inline.hpp
index 1b4664f5a4b..f9db34f4884 100644
--- a/src/hotspot/share/oops/instanceKlass.inline.hpp
+++ b/src/hotspot/share/oops/instanceKlass.inline.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2015, 2023, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2015, 2025, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -71,14 +71,6 @@ inline void InstanceKlass::release_set_array_klasses(ObjArrayKlass* k) {
Atomic::release_store(&_array_klasses, k);
}
-inline jmethodID* InstanceKlass::methods_jmethod_ids_acquire() const {
- return Atomic::load_acquire(&_methods_jmethod_ids);
-}
-
-inline void InstanceKlass::release_set_methods_jmethod_ids(jmethodID* jmeths) {
- Atomic::release_store(&_methods_jmethod_ids, jmeths);
-}
-
// The iteration over the oops in objects is a hot path in the GC code.
// By force inlining the following functions, we get similar GC performance
// as the previous macro based implementation.
diff --git a/src/hotspot/share/oops/jmethodIDTable.cpp b/src/hotspot/share/oops/jmethodIDTable.cpp
new file mode 100644
index 00000000000..9dba79229e7
--- /dev/null
+++ b/src/hotspot/share/oops/jmethodIDTable.cpp
@@ -0,0 +1,194 @@
+/*
+ * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ *
+ */
+
+#include "logging/log.hpp"
+#include "memory/allocation.hpp"
+#include "memory/resourceArea.hpp"
+#include "oops/jmethodIDTable.hpp"
+#include "oops/method.hpp"
+#include "runtime/mutexLocker.hpp"
+#include "utilities/concurrentHashTable.inline.hpp"
+#include "utilities/concurrentHashTableTasks.inline.hpp"
+#include "utilities/macros.hpp"
+
+// Save (jmethod, Method*) in a hashtable to lookup Method.
+// The CHT is for performance because it has lock free lookup.
+// The value of the next jmethodID. This only increments (always unique IDs).
+static uint64_t _jmethodID_counter = 0;
+// Tracks the number of jmethodID entries in the _jmethod_id_table.
+// Incremented on insert, decremented on remove. Used to track if we need to resize the table.
+static uint64_t _jmethodID_entry_count = 0;
+
+uint64_t JmethodIDTable::get_entry_count() { return _jmethodID_entry_count; }
+
+class JmethodEntry {
+ public:
+ uint64_t _id;
+ Method* _method;
+
+ JmethodEntry(uint64_t id, Method* method) : _id(id), _method(method) {}
+};
+
+class JmethodIDTableConfig : public AllStatic {
+ public:
+ typedef JmethodEntry Value;
+ static void* allocate_node(void* context, size_t size, Value const& value) {
+ return AllocateHeap(size, mtJNI);
+ }
+ static void free_node(void* context, void* memory, Value const& value) {
+ FreeHeap(memory);
+ }
+ static uintx get_hash(Value const& value, bool* is_dead) {
+ *is_dead = false;
+ return value._id;
+ }
+ static bool is_dead(Value const& value) {
+ return false;
+ }
+};
+
+using MethodIdTable = ConcurrentHashTable;
+static MethodIdTable* _jmethod_id_table = nullptr;
+
+void JmethodIDTable::initialize() {
+ const size_t start_size = 10;
+ // 2^24 is max size
+ const size_t end_size = 24;
+ // If a chain gets to 32 something might be wrong.
+ const size_t grow_hint = 32;
+
+ _jmethod_id_table = new MethodIdTable(start_size, end_size, grow_hint);
+}
+
+class JmethodIDLookup : StackObj {
+ private:
+ uint64_t _mid;
+
+ public:
+ JmethodIDLookup(const uint64_t mid) : _mid(mid) {}
+ uintx get_hash() const {
+ return _mid;
+ }
+ bool equals(JmethodEntry* value) {
+ return _mid == value->_id;
+ }
+
+ bool is_dead(JmethodEntry* value) {
+ return false;
+ }
+};
+
+static JmethodEntry* get_jmethod_entry(jmethodID mid) {
+ assert(mid != nullptr, "JNI method id should not be null");
+
+ Thread* current = Thread::current();
+ JmethodIDLookup lookup((uint64_t)mid);
+ JmethodEntry* result = nullptr;
+ auto get = [&] (JmethodEntry* value) {
+ // function called if value is found so is never null
+ result = value;
+ };
+ bool needs_rehashing = false;
+ _jmethod_id_table->get(current, lookup, get, &needs_rehashing);
+ assert(!needs_rehashing, "should never need rehashing");
+ return result;
+}
+
+Method* JmethodIDTable::resolve_jmethod_id(jmethodID mid) {
+ JmethodEntry* result = get_jmethod_entry(mid);
+ return result == nullptr ? nullptr : result->_method;
+}
+
+const unsigned _resize_load_trigger = 5; // load factor that will trigger the resize
+
+static unsigned table_size(Thread* current) {
+ return 1 << _jmethod_id_table->get_size_log2(current);
+}
+
+static bool needs_resize(Thread* current) {
+ return ((_jmethodID_entry_count > (_resize_load_trigger * table_size(current))) &&
+ !_jmethod_id_table->is_max_size_reached());
+}
+
+// Add a method id to the jmethod_ids.
+jmethodID JmethodIDTable::make_jmethod_id(Method* m) {
+ bool grow_hint, clean_hint;
+
+ assert_locked_or_safepoint(JmethodIdCreation_lock);
+ // Update jmethodID global counter.
+ _jmethodID_counter++;
+
+ JmethodEntry new_entry(_jmethodID_counter, m);
+ Thread* current = Thread::current();
+ JmethodIDLookup lookup(_jmethodID_counter);
+ bool created = _jmethod_id_table->insert(current, lookup, new_entry, &grow_hint, &clean_hint);
+ assert(created, "must be");
+ log_debug(jmethod)("Inserted jmethod id " UINT64_FORMAT_X, _jmethodID_counter);
+ // Increment number of entries in the table.
+ _jmethodID_entry_count++;
+
+ // Resize table if it needs to grow. The _jmethod_id_table has a good distribution.
+ if (needs_resize(current)) {
+ _jmethod_id_table->grow(current);
+ log_info(jmethod)("Growing table to %d for " UINT64_FORMAT " entries", table_size(current), _jmethodID_counter);
+ }
+ return (jmethodID)_jmethodID_counter;
+}
+
+void JmethodIDTable::remove(jmethodID jmid) {
+ assert_locked_or_safepoint(JmethodIdCreation_lock);
+ JmethodIDLookup lookup((uint64_t)jmid);
+ Thread* current = Thread::current();
+ JmethodEntry* result;
+ auto get = [&] (JmethodEntry* value) {
+ // The function that is called if value is found, so is never null.
+ result = value;
+ };
+ bool removed = _jmethod_id_table->remove(current, lookup, get);
+ assert(removed, "must be");
+ log_debug(jmethod)("Removed jmethod id " UINT64_FORMAT_X, (uint64_t)jmid);
+ // Decrement number of entries in the table.
+ _jmethodID_entry_count--;
+}
+
+void JmethodIDTable::change_method_associated_with_jmethod_id(jmethodID jmid, Method* new_method) {
+ assert_locked_or_safepoint(JmethodIdCreation_lock);
+ JmethodEntry* result = get_jmethod_entry(jmid);
+ // Change table entry to point to the new method.
+ log_debug(jmethod)("Changed jmethod id " UINT64_FORMAT_X " from " PTR_FORMAT " to " PTR_FORMAT, (uint64_t)jmid,
+ p2i(result->_method), p2i(new_method));
+ result->_method = new_method;
+}
+
+void JmethodIDTable::clear_jmethod_id(jmethodID jmid, Method* obsolete_method) {
+ assert_locked_or_safepoint(JmethodIdCreation_lock);
+ JmethodEntry* result = get_jmethod_entry(jmid);
+ // We need to make sure that jmethodID actually resolves to this method
+ // - multiple redefined versions may share jmethodID slots and if a method
+ // has already been rewired to a newer version we could be clearing
+ // the reference to a still existing method instance.
+ if (result->_method == obsolete_method) {
+ result->_method = nullptr;
+ }
+}
diff --git a/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/LockingMode.java b/src/hotspot/share/oops/jmethodIDTable.hpp
similarity index 51%
rename from src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/LockingMode.java
rename to src/hotspot/share/oops/jmethodIDTable.hpp
index 2046fd075ad..cdbe5c87855 100644
--- a/src/jdk.hotspot.agent/share/classes/sun/jvm/hotspot/runtime/LockingMode.java
+++ b/src/hotspot/share/oops/jmethodIDTable.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+ * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -22,39 +22,34 @@
*
*/
-package sun.jvm.hotspot.runtime;
+#ifndef SHARE_OOPS_JMETHODIDTABLE_HPP
+#define SHARE_OOPS_JMETHODIDTABLE_HPP
-import sun.jvm.hotspot.types.TypeDataBase;
+#include "jni.h"
+#include "memory/allocation.hpp"
+// Class for associating Method with jmethodID
+class Method;
-/** Encapsulates the LockingMode enum in globalDefinitions.hpp in
- the VM. */
+class JmethodIDTable : public AllStatic {
+ public:
+ static void initialize();
-public class LockingMode {
- private static int monitor;
- private static int legacy;
- private static int lightweight;
+ // Given a Method return a jmethodID.
+ static jmethodID make_jmethod_id(Method* m);
- static {
- VM.registerVMInitializedObserver(
- (o, d) -> initialize(VM.getVM().getTypeDataBase()));
- }
+ // Given a jmethodID, return a Method.
+ static Method* resolve_jmethod_id(jmethodID mid);
- private static synchronized void initialize(TypeDataBase db) {
- monitor = db.lookupIntConstant("LM_MONITOR").intValue();
- legacy = db.lookupIntConstant("LM_LEGACY").intValue();
- lightweight = db.lookupIntConstant("LM_LIGHTWEIGHT").intValue();
- }
+ // Class unloading support, remove the associations from the tables. Stale jmethodID will
+ // not be found and return null.
+ static void remove(jmethodID mid);
- public static int getMonitor() {
- return monitor;
- }
+ // RedefineClasses support
+ static void change_method_associated_with_jmethod_id(jmethodID jmid, Method* new_method);
+ static void clear_jmethod_id(jmethodID jmid, Method* m);
- public static int getLegacy() {
- return legacy;
- }
+ static uint64_t get_entry_count();
+};
- public static int getLightweight() {
- return lightweight;
- }
-}
+#endif // SHARE_OOPS_JMETHODIDTABLE_HPP
diff --git a/src/hotspot/share/oops/method.cpp b/src/hotspot/share/oops/method.cpp
index 1a36fce23aa..7552bf50ed9 100644
--- a/src/hotspot/share/oops/method.cpp
+++ b/src/hotspot/share/oops/method.cpp
@@ -53,6 +53,7 @@
#include "nmt/memTracker.hpp"
#include "oops/constMethod.hpp"
#include "oops/constantPool.hpp"
+#include "oops/jmethodIDTable.hpp"
#include "oops/klass.inline.hpp"
#include "oops/method.inline.hpp"
#include "oops/methodData.hpp"
@@ -1028,7 +1029,7 @@ void Method::set_native_function(address function, bool post_event_flag) {
// If so, we have to make it not_entrant.
nmethod* nm = code(); // Put it into local variable to guard against concurrent updates
if (nm != nullptr) {
- nm->make_not_entrant(nmethod::ChangeReason::set_native_function);
+ nm->make_not_entrant(nmethod::InvalidationReason::SET_NATIVE_FUNCTION);
}
}
@@ -2059,172 +2060,33 @@ void BreakpointInfo::clear(Method* method) {
#endif // INCLUDE_JVMTI
// jmethodID handling
-
-// This is a block allocating object, sort of like JNIHandleBlock, only a
-// lot simpler.
-// It's allocated on the CHeap because once we allocate a jmethodID, we can
-// never get rid of it.
-
-static const int min_block_size = 8;
-
-class JNIMethodBlockNode : public CHeapObj {
- friend class JNIMethodBlock;
- Method** _methods;
- int _number_of_methods;
- int _top;
- JNIMethodBlockNode* _next;
-
- public:
-
- JNIMethodBlockNode(int num_methods = min_block_size);
-
- ~JNIMethodBlockNode() { FREE_C_HEAP_ARRAY(Method*, _methods); }
-
- void ensure_methods(int num_addl_methods) {
- if (_top < _number_of_methods) {
- num_addl_methods -= _number_of_methods - _top;
- if (num_addl_methods <= 0) {
- return;
- }
- }
- if (_next == nullptr) {
- _next = new JNIMethodBlockNode(MAX2(num_addl_methods, min_block_size));
- } else {
- _next->ensure_methods(num_addl_methods);
- }
- }
-};
-
-class JNIMethodBlock : public CHeapObj {
- JNIMethodBlockNode _head;
- JNIMethodBlockNode *_last_free;
- public:
- static Method* const _free_method;
-
- JNIMethodBlock(int initial_capacity = min_block_size)
- : _head(initial_capacity), _last_free(&_head) {}
-
- void ensure_methods(int num_addl_methods) {
- _last_free->ensure_methods(num_addl_methods);
- }
-
- Method** add_method(Method* m) {
- for (JNIMethodBlockNode* b = _last_free; b != nullptr; b = b->_next) {
- if (b->_top < b->_number_of_methods) {
- // top points to the next free entry.
- int i = b->_top;
- b->_methods[i] = m;
- b->_top++;
- _last_free = b;
- return &(b->_methods[i]);
- } else if (b->_top == b->_number_of_methods) {
- // if the next free entry ran off the block see if there's a free entry
- for (int i = 0; i < b->_number_of_methods; i++) {
- if (b->_methods[i] == _free_method) {
- b->_methods[i] = m;
- _last_free = b;
- return &(b->_methods[i]);
- }
- }
- // Only check each block once for frees. They're very unlikely.
- // Increment top past the end of the block.
- b->_top++;
- }
- // need to allocate a next block.
- if (b->_next == nullptr) {
- b->_next = _last_free = new JNIMethodBlockNode();
- }
- }
- guarantee(false, "Should always allocate a free block");
- return nullptr;
- }
-
- bool contains(Method** m) {
- if (m == nullptr) return false;
- for (JNIMethodBlockNode* b = &_head; b != nullptr; b = b->_next) {
- if (b->_methods <= m && m < b->_methods + b->_number_of_methods) {
- // This is a bit of extra checking, for two reasons. One is
- // that contains() deals with pointers that are passed in by
- // JNI code, so making sure that the pointer is aligned
- // correctly is valuable. The other is that <= and > are
- // technically not defined on pointers, so the if guard can
- // pass spuriously; no modern compiler is likely to make that
- // a problem, though (and if one did, the guard could also
- // fail spuriously, which would be bad).
- ptrdiff_t idx = m - b->_methods;
- if (b->_methods + idx == m) {
- return true;
- }
- }
- }
- return false; // not found
- }
-
- // During class unloading the methods are cleared, which is different
- // than freed.
- void clear_all_methods() {
- for (JNIMethodBlockNode* b = &_head; b != nullptr; b = b->_next) {
- for (int i = 0; i< b->_number_of_methods; i++) {
- b->_methods[i] = nullptr;
- }
- }
- }
-#ifndef PRODUCT
- int count_methods() {
- // count all allocated methods
- int count = 0;
- for (JNIMethodBlockNode* b = &_head; b != nullptr; b = b->_next) {
- for (int i = 0; i< b->_number_of_methods; i++) {
- if (b->_methods[i] != _free_method) count++;
- }
- }
- return count;
- }
-#endif // PRODUCT
-};
-
-// Something that can't be mistaken for an address or a markWord
-Method* const JNIMethodBlock::_free_method = (Method*)55;
-
-JNIMethodBlockNode::JNIMethodBlockNode(int num_methods) : _top(0), _next(nullptr) {
- _number_of_methods = MAX2(num_methods, min_block_size);
- _methods = NEW_C_HEAP_ARRAY(Method*, _number_of_methods, mtInternal);
- for (int i = 0; i < _number_of_methods; i++) {
- _methods[i] = JNIMethodBlock::_free_method;
- }
-}
-
-void Method::ensure_jmethod_ids(ClassLoaderData* cld, int capacity) {
- // Have to add jmethod_ids() to class loader data thread-safely.
- // Also have to add the method to the list safely, which the lock
- // protects as well.
- MutexLocker ml(JmethodIdCreation_lock, Mutex::_no_safepoint_check_flag);
- if (cld->jmethod_ids() == nullptr) {
- cld->set_jmethod_ids(new JNIMethodBlock(capacity));
- } else {
- cld->jmethod_ids()->ensure_methods(capacity);
- }
-}
+// jmethodIDs are 64-bit integers that will never run out and are mapped in a table
+// to their Method and vice versa. If JNI code has access to stale jmethodID, this
+// wastes no memory but the Method* returned is null.
// Add a method id to the jmethod_ids
jmethodID Method::make_jmethod_id(ClassLoaderData* cld, Method* m) {
// Have to add jmethod_ids() to class loader data thread-safely.
- // Also have to add the method to the list safely, which the lock
+ // Also have to add the method to the InstanceKlass list safely, which the lock
// protects as well.
assert(JmethodIdCreation_lock->owned_by_self(), "sanity check");
+ jmethodID jmid = JmethodIDTable::make_jmethod_id(m);
+ assert(jmid != nullptr, "must be created");
- ResourceMark rm;
- log_debug(jmethod)("Creating jmethodID for Method %s", m->external_name());
- if (cld->jmethod_ids() == nullptr) {
- cld->set_jmethod_ids(new JNIMethodBlock());
- }
- // jmethodID is a pointer to Method*
- return (jmethodID)cld->jmethod_ids()->add_method(m);
+ // Add to growable array in CLD.
+ cld->add_jmethod_id(jmid);
+ return jmid;
}
+// This looks in the InstanceKlass cache, then calls back to make_jmethod_id if not found.
jmethodID Method::jmethod_id() {
- methodHandle mh(Thread::current(), this);
- return method_holder()->get_jmethod_id(mh);
+ return method_holder()->get_jmethod_id(this);
+}
+
+// Get the Method out of the table given the method id.
+Method* Method::resolve_jmethod_id(jmethodID mid) {
+ assert(mid != nullptr, "JNI method id should not be null");
+ return JmethodIDTable::resolve_jmethod_id(mid);
}
void Method::change_method_associated_with_jmethod_id(jmethodID jmid, Method* new_method) {
@@ -2232,25 +2094,34 @@ void Method::change_method_associated_with_jmethod_id(jmethodID jmid, Method* ne
// scratch method holder.
assert(resolve_jmethod_id(jmid)->method_holder()->class_loader()
== new_method->method_holder()->class_loader() ||
- new_method->method_holder()->class_loader() == nullptr, // allow Unsafe substitution
+ new_method->method_holder()->class_loader() == nullptr, // allow substitution to Unsafe method
"changing to a different class loader");
- // Just change the method in place, jmethodID pointer doesn't change.
- *((Method**)jmid) = new_method;
+ JmethodIDTable::change_method_associated_with_jmethod_id(jmid, new_method);
}
-bool Method::is_method_id(jmethodID mid) {
+// If there's a jmethodID for this method, clear the Method
+// but leave jmethodID for this method in the table.
+// It's deallocated with class unloading.
+void Method::clear_jmethod_id() {
+ jmethodID mid = method_holder()->jmethod_id_or_null(this);
+ if (mid != nullptr) {
+ JmethodIDTable::clear_jmethod_id(mid, this);
+ }
+}
+
+bool Method::validate_jmethod_id(jmethodID mid) {
Method* m = resolve_jmethod_id(mid);
assert(m != nullptr, "should be called with non-null method");
InstanceKlass* ik = m->method_holder();
ClassLoaderData* cld = ik->class_loader_data();
if (cld->jmethod_ids() == nullptr) return false;
- return (cld->jmethod_ids()->contains((Method**)mid));
+ return (cld->jmethod_ids()->contains(mid));
}
Method* Method::checked_resolve_jmethod_id(jmethodID mid) {
if (mid == nullptr) return nullptr;
Method* o = resolve_jmethod_id(mid);
- if (o == nullptr || o == JNIMethodBlock::_free_method) {
+ if (o == nullptr) {
return nullptr;
}
// Method should otherwise be valid. Assert for testing.
@@ -2259,7 +2130,7 @@ Method* Method::checked_resolve_jmethod_id(jmethodID mid) {
// unloaded, we need to return null here too because after a safepoint, its memory
// will be reclaimed.
return o->method_holder()->is_loader_alive() ? o : nullptr;
-};
+}
void Method::set_on_stack(const bool value) {
// Set both the method itself and its constant pool. The constant pool
@@ -2280,25 +2151,6 @@ void Method::record_gc_epoch() {
constants()->cache()->record_gc_epoch();
}
-// Called when the class loader is unloaded to make all methods weak.
-void Method::clear_jmethod_ids(ClassLoaderData* loader_data) {
- loader_data->jmethod_ids()->clear_all_methods();
-}
-
-void Method::clear_jmethod_id() {
- // Being at a safepoint prevents racing against other class redefinitions
- assert(SafepointSynchronize::is_at_safepoint(), "should be at safepoint");
- // The jmethodID is not stored in the Method instance, we need to look it up first
- jmethodID methodid = find_jmethod_id_or_null();
- // We need to make sure that jmethodID actually resolves to this method
- // - multiple redefined versions may share jmethodID slots and if a method
- // has already been rewired to a newer version we could be removing reference
- // to a still existing method instance
- if (methodid != nullptr && *((Method**)methodid) == this) {
- *((Method**)methodid) = nullptr;
- }
-}
-
bool Method::has_method_vptr(const void* ptr) {
Method m;
// This assumes that the vtbl pointer is the first word of a C++ object.
@@ -2323,13 +2175,6 @@ bool Method::is_valid_method(const Method* m) {
}
}
-#ifndef PRODUCT
-void Method::print_jmethod_ids_count(const ClassLoaderData* loader_data, outputStream* out) {
- out->print("%d", loader_data->jmethod_ids()->count_methods());
-}
-#endif // PRODUCT
-
-
// Printing
#ifndef PRODUCT
diff --git a/src/hotspot/share/oops/method.hpp b/src/hotspot/share/oops/method.hpp
index ce6b859f7c1..b241104b62c 100644
--- a/src/hotspot/share/oops/method.hpp
+++ b/src/hotspot/share/oops/method.hpp
@@ -704,18 +704,10 @@ class Method : public Metadata {
// refers to null (as is the case for any weak reference).
static jmethodID make_jmethod_id(ClassLoaderData* cld, Method* mh);
- // Ensure there is enough capacity in the internal tracking data
- // structures to hold the number of jmethodIDs you plan to generate.
- // This saves substantial time doing allocations.
- static void ensure_jmethod_ids(ClassLoaderData* cld, int capacity);
-
// Use resolve_jmethod_id() in situations where the caller is expected
// to provide a valid jmethodID; the only sanity checks are in asserts;
// result guaranteed not to be null.
- inline static Method* resolve_jmethod_id(jmethodID mid) {
- assert(mid != nullptr, "JNI method id should not be null");
- return *((Method**)mid);
- }
+ static Method* resolve_jmethod_id(jmethodID mid);
// Use checked_resolve_jmethod_id() in situations where the caller
// should provide a valid jmethodID, but might not. Null is returned
@@ -723,10 +715,9 @@ class Method : public Metadata {
static Method* checked_resolve_jmethod_id(jmethodID mid);
static void change_method_associated_with_jmethod_id(jmethodID old_jmid_ptr, Method* new_method);
- static bool is_method_id(jmethodID mid);
+ static bool validate_jmethod_id(jmethodID mid);
- // Clear methods
- static void clear_jmethod_ids(ClassLoaderData* loader_data);
+ // Clear jmethodID
void clear_jmethod_id();
static void print_jmethod_ids_count(const ClassLoaderData* loader_data, outputStream* out) PRODUCT_RETURN;
diff --git a/src/hotspot/share/oops/trainingData.cpp b/src/hotspot/share/oops/trainingData.cpp
index 2ae1ac18699..cc15fe944f2 100644
--- a/src/hotspot/share/oops/trainingData.cpp
+++ b/src/hotspot/share/oops/trainingData.cpp
@@ -220,8 +220,8 @@ CompileTrainingData* CompileTrainingData::make(CompileTask* task) {
last_ctd = ctd;
}
} else {
- last_ctd = ctd;
- mtd->notice_toplevel_compilation(level);
+ last_ctd = ctd;
+ mtd->notice_toplevel_compilation(level);
}
}
return ctd;
diff --git a/src/hotspot/share/oops/trainingData.hpp b/src/hotspot/share/oops/trainingData.hpp
index 2be3026881a..6acf19b4d5e 100644
--- a/src/hotspot/share/oops/trainingData.hpp
+++ b/src/hotspot/share/oops/trainingData.hpp
@@ -34,6 +34,7 @@
#include "memory/metaspaceClosure.hpp"
#include "oops/instanceKlass.hpp"
#include "oops/method.hpp"
+#include "oops/objArrayKlass.hpp"
#include "runtime/handles.hpp"
#include "runtime/mutexLocker.hpp"
#include "utilities/resizeableResourceHash.hpp"
@@ -286,7 +287,12 @@ class TrainingData : public Metadata {
static bool is_klass_loaded(Klass* k) {
if (have_data()) {
// If we're running in AOT mode some classes may not be loaded yet
- return !k->is_instance_klass() || InstanceKlass::cast(k)->is_loaded();
+ if (k->is_objArray_klass()) {
+ k = ObjArrayKlass::cast(k)->bottom_klass();
+ }
+ if (k->is_instance_klass()) {
+ return InstanceKlass::cast(k)->is_loaded();
+ }
}
return true;
}
diff --git a/src/hotspot/share/opto/addnode.cpp b/src/hotspot/share/opto/addnode.cpp
index 143398c57d0..92fb54a3a13 100644
--- a/src/hotspot/share/opto/addnode.cpp
+++ b/src/hotspot/share/opto/addnode.cpp
@@ -32,8 +32,8 @@
#include "opto/mulnode.hpp"
#include "opto/phaseX.hpp"
#include "opto/subnode.hpp"
-#include "runtime/stubRoutines.hpp"
#include "opto/utilities/xor.hpp"
+#include "runtime/stubRoutines.hpp"
// Portions of code courtesy of Clifford Click
diff --git a/src/hotspot/share/opto/block.cpp b/src/hotspot/share/opto/block.cpp
index c33f656047d..7d3d4ec16f4 100644
--- a/src/hotspot/share/opto/block.cpp
+++ b/src/hotspot/share/opto/block.cpp
@@ -22,10 +22,10 @@
*
*/
+#include "compiler/compilerDirectives.hpp"
#include "libadt/vectset.hpp"
#include "memory/allocation.inline.hpp"
#include "memory/resourceArea.hpp"
-#include "compiler/compilerDirectives.hpp"
#include "opto/block.hpp"
#include "opto/cfgnode.hpp"
#include "opto/chaitin.hpp"
@@ -37,11 +37,8 @@
#include "utilities/copy.hpp"
#include "utilities/powerOfTwo.hpp"
-void Block_Array::grow( uint i ) {
- _nesting.check(_arena); // Check if a potential reallocation in the arena is safe
- if (i < Max()) {
- return; // No need to grow
- }
+void Block_Array::grow(uint i) {
+ assert(i >= Max(), "Should have been checked before, use maybe_grow?");
DEBUG_ONLY(_limit = i+1);
if( i < _size ) return;
if( !_size ) {
diff --git a/src/hotspot/share/opto/block.hpp b/src/hotspot/share/opto/block.hpp
index e6a98c28e77..07faca08dc3 100644
--- a/src/hotspot/share/opto/block.hpp
+++ b/src/hotspot/share/opto/block.hpp
@@ -53,7 +53,13 @@ class Block_Array : public ArenaObj {
ReallocMark _nesting; // Safety checks for arena reallocation
protected:
Block **_blocks;
- void grow( uint i ); // Grow array node to fit
+ void maybe_grow(uint i) {
+ _nesting.check(_arena); // Check if a potential reallocation in the arena is safe
+ if (i >= Max()) {
+ grow(i);
+ }
+ }
+ void grow(uint i); // Grow array node to fit
public:
Block_Array(Arena *a) : _size(OptoBlockListSize), _arena(a) {
@@ -68,7 +74,7 @@ class Block_Array : public ArenaObj {
Block *operator[] ( uint i ) const // Lookup, or assert for not mapped
{ assert( i < Max(), "oob" ); return _blocks[i]; }
// Extend the mapping: index i maps to Block *n.
- void map( uint i, Block *n ) { grow(i); _blocks[i] = n; }
+ void map( uint i, Block *n ) { maybe_grow(i); _blocks[i] = n; }
uint Max() const { DEBUG_ONLY(return _limit); return _size; }
};
diff --git a/src/hotspot/share/opto/bytecodeInfo.cpp b/src/hotspot/share/opto/bytecodeInfo.cpp
index e618a708f61..547cf2f6a38 100644
--- a/src/hotspot/share/opto/bytecodeInfo.cpp
+++ b/src/hotspot/share/opto/bytecodeInfo.cpp
@@ -26,8 +26,8 @@
#include "classfile/vmSymbols.hpp"
#include "compiler/compilationPolicy.hpp"
#include "compiler/compileBroker.hpp"
-#include "compiler/compilerEvent.hpp"
#include "compiler/compileLog.hpp"
+#include "compiler/compilerEvent.hpp"
#include "interpreter/linkResolver.hpp"
#include "jfr/jfrEvents.hpp"
#include "oops/objArrayKlass.hpp"
diff --git a/src/hotspot/share/opto/c2_globals.hpp b/src/hotspot/share/opto/c2_globals.hpp
index fd55f2fd666..ca76114af31 100644
--- a/src/hotspot/share/opto/c2_globals.hpp
+++ b/src/hotspot/share/opto/c2_globals.hpp
@@ -25,9 +25,9 @@
#ifndef SHARE_OPTO_C2_GLOBALS_HPP
#define SHARE_OPTO_C2_GLOBALS_HPP
-#include "opto/c2_globals_pd.hpp"
#include "runtime/globals_shared.hpp"
#include "utilities/macros.hpp"
+#include CPU_HEADER(c2_globals)
//
// Defines all globals flags used by the server compiler.
@@ -678,10 +678,12 @@
"Print progress during Iterative Global Value Numbering") \
\
develop(uint, VerifyIterativeGVN, 0, \
- "Verify Iterative Global Value Numbering" \
- "=XY, with Y: verify Def-Use modifications during IGVN" \
- " X: verify that type(n) == n->Value() after IGVN" \
- "X and Y in 0=off; 1=on") \
+ "Verify Iterative Global Value Numbering =DCBA, with:" \
+ " D: verify Node::Identity did not miss opportunities" \
+ " C: verify Node::Ideal did not miss opportunities" \
+ " B: verify that type(n) == n->Value() after IGVN" \
+ " A: verify Def-Use modifications during IGVN" \
+ "Each can be 0=off or 1=on") \
constraint(VerifyIterativeGVNConstraintFunc, AtParse) \
\
develop(bool, TraceCISCSpill, false, \
diff --git a/src/hotspot/share/opto/c2compiler.cpp b/src/hotspot/share/opto/c2compiler.cpp
index 0c642211e1f..e5b1e131505 100644
--- a/src/hotspot/share/opto/c2compiler.cpp
+++ b/src/hotspot/share/opto/c2compiler.cpp
@@ -25,15 +25,15 @@
#include "classfile/vmClasses.hpp"
#include "compiler/compilationMemoryStatistic.hpp"
#include "compiler/compilerDefinitions.inline.hpp"
-#include "runtime/handles.inline.hpp"
#include "jfr/support/jfrIntrinsics.hpp"
#include "opto/c2compiler.hpp"
#include "opto/compile.hpp"
#include "opto/optoreg.hpp"
#include "opto/output.hpp"
#include "opto/runtime.hpp"
-#include "runtime/stubRoutines.hpp"
#include "runtime/globals_extension.hpp"
+#include "runtime/handles.inline.hpp"
+#include "runtime/stubRoutines.hpp"
#include "utilities/macros.hpp"
@@ -767,7 +767,7 @@ bool C2Compiler::is_intrinsic_supported(vmIntrinsics::ID id) {
case vmIntrinsics::_doubleToRawLongBits:
case vmIntrinsics::_doubleToLongBits:
case vmIntrinsics::_longBitsToDouble:
- case vmIntrinsics::_Reference_get:
+ case vmIntrinsics::_Reference_get0:
case vmIntrinsics::_Reference_refersTo0:
case vmIntrinsics::_PhantomReference_refersTo0:
case vmIntrinsics::_Reference_clear0:
diff --git a/src/hotspot/share/opto/callGenerator.cpp b/src/hotspot/share/opto/callGenerator.cpp
index e0ad2f00ab5..e09d8cabe2c 100644
--- a/src/hotspot/share/opto/callGenerator.cpp
+++ b/src/hotspot/share/opto/callGenerator.cpp
@@ -24,9 +24,9 @@
#include "ci/bcEscapeAnalyzer.hpp"
#include "ci/ciCallSite.hpp"
-#include "ci/ciObjArray.hpp"
#include "ci/ciMemberName.hpp"
#include "ci/ciMethodHandle.hpp"
+#include "ci/ciObjArray.hpp"
#include "classfile/javaClasses.hpp"
#include "compiler/compileLog.hpp"
#include "opto/addnode.hpp"
diff --git a/src/hotspot/share/opto/callnode.cpp b/src/hotspot/share/opto/callnode.cpp
index 3527e46c24b..a6203c94cff 100644
--- a/src/hotspot/share/opto/callnode.cpp
+++ b/src/hotspot/share/opto/callnode.cpp
@@ -22,8 +22,9 @@
*
*/
-#include "compiler/compileLog.hpp"
#include "ci/bcEscapeAnalyzer.hpp"
+#include "code/vmreg.hpp"
+#include "compiler/compileLog.hpp"
#include "compiler/oopMap.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/c2/barrierSetC2.hpp"
@@ -43,7 +44,6 @@
#include "opto/runtime.hpp"
#include "runtime/sharedRuntime.hpp"
#include "utilities/powerOfTwo.hpp"
-#include "code/vmreg.hpp"
// Portions of code courtesy of Clifford Click
diff --git a/src/hotspot/share/opto/castnode.cpp b/src/hotspot/share/opto/castnode.cpp
index 6b3058d94f6..96f5ba7e693 100644
--- a/src/hotspot/share/opto/castnode.cpp
+++ b/src/hotspot/share/opto/castnode.cpp
@@ -22,6 +22,7 @@
*
*/
+#include "castnode.hpp"
#include "opto/addnode.hpp"
#include "opto/callnode.hpp"
#include "opto/castnode.hpp"
@@ -30,7 +31,6 @@
#include "opto/phaseX.hpp"
#include "opto/subnode.hpp"
#include "opto/type.hpp"
-#include "castnode.hpp"
#include "utilities/checkedCast.hpp"
//=============================================================================
@@ -498,7 +498,11 @@ Node* ConstraintCastNode::make_cast_for_type(Node* c, Node* in, const Type* type
Node* ConstraintCastNode::optimize_integer_cast(PhaseGVN* phase, BasicType bt) {
PhaseIterGVN *igvn = phase->is_IterGVN();
- const TypeInteger* this_type = this->type()->is_integer(bt);
+ const TypeInteger* this_type = this->type()->isa_integer(bt);
+ if (this_type == nullptr) {
+ return nullptr;
+ }
+
Node* z = in(1);
const TypeInteger* rx = nullptr;
const TypeInteger* ry = nullptr;
diff --git a/src/hotspot/share/opto/cfgnode.cpp b/src/hotspot/share/opto/cfgnode.cpp
index 92f6c938dba..ef912ff471a 100644
--- a/src/hotspot/share/opto/cfgnode.cpp
+++ b/src/hotspot/share/opto/cfgnode.cpp
@@ -35,8 +35,8 @@
#include "opto/loopnode.hpp"
#include "opto/machnode.hpp"
#include "opto/movenode.hpp"
-#include "opto/narrowptrnode.hpp"
#include "opto/mulnode.hpp"
+#include "opto/narrowptrnode.hpp"
#include "opto/phaseX.hpp"
#include "opto/regalloc.hpp"
#include "opto/regmask.hpp"
diff --git a/src/hotspot/share/opto/classes.cpp b/src/hotspot/share/opto/classes.cpp
index 61d164773a2..b760a179b57 100644
--- a/src/hotspot/share/opto/classes.cpp
+++ b/src/hotspot/share/opto/classes.cpp
@@ -35,8 +35,8 @@
#include "opto/locknode.hpp"
#include "opto/loopnode.hpp"
#include "opto/machnode.hpp"
-#include "opto/memnode.hpp"
#include "opto/mathexactnode.hpp"
+#include "opto/memnode.hpp"
#include "opto/movenode.hpp"
#include "opto/mulnode.hpp"
#include "opto/multnode.hpp"
diff --git a/src/hotspot/share/opto/compile.cpp b/src/hotspot/share/opto/compile.cpp
index a5023408cdf..4c5f382ceee 100644
--- a/src/hotspot/share/opto/compile.cpp
+++ b/src/hotspot/share/opto/compile.cpp
@@ -33,9 +33,9 @@
#include "compiler/compilationMemoryStatistic.hpp"
#include "compiler/compileBroker.hpp"
#include "compiler/compileLog.hpp"
+#include "compiler/compiler_globals.hpp"
#include "compiler/compilerDefinitions.hpp"
#include "compiler/compilerOracle.hpp"
-#include "compiler/compiler_globals.hpp"
#include "compiler/disassembler.hpp"
#include "compiler/oopMap.hpp"
#include "gc/shared/barrierSet.hpp"
@@ -783,19 +783,9 @@ Compile::Compile(ciEnv* ci_env, ciMethod* target, int osr_bci,
StartNode* s = new StartNode(root(), tf()->domain());
initial_gvn()->set_type_bottom(s);
verify_start(s);
- if (method()->intrinsic_id() == vmIntrinsics::_Reference_get) {
- // With java.lang.ref.reference.get() we must go through the
- // intrinsic - even when get() is the root
- // method of the compile - so that, if necessary, the value in
- // the referent field of the reference object gets recorded by
- // the pre-barrier code.
- cg = find_intrinsic(method(), false);
- }
- if (cg == nullptr) {
- float past_uses = method()->interpreter_invocation_count();
- float expected_uses = past_uses;
- cg = CallGenerator::for_inline(method(), expected_uses);
- }
+ float past_uses = method()->interpreter_invocation_count();
+ float expected_uses = past_uses;
+ cg = CallGenerator::for_inline(method(), expected_uses);
}
if (failing()) return;
if (cg == nullptr) {
@@ -4521,7 +4511,9 @@ Node* Compile::conv_I2X_index(PhaseGVN* phase, Node* idx, const TypeInt* sizetyp
// number. (The prior range check has ensured this.)
// This assertion is used by ConvI2LNode::Ideal.
int index_max = max_jint - 1; // array size is max_jint, index is one less
- if (sizetype != nullptr) index_max = sizetype->_hi - 1;
+ if (sizetype != nullptr && sizetype->_hi > 0) {
+ index_max = sizetype->_hi - 1;
+ }
const TypeInt* iidxtype = TypeInt::make(0, index_max, Type::WidenMax);
idx = constrained_convI2L(phase, idx, iidxtype, ctrl);
#endif
@@ -4549,7 +4541,7 @@ void Compile::dump_print_inlining() {
void Compile::log_late_inline(CallGenerator* cg) {
if (log() != nullptr) {
- log()->head("late_inline method='%d' inline_id='" JLONG_FORMAT "'", log()->identify(cg->method()),
+ log()->head("late_inline method='%d' inline_id='" JLONG_FORMAT "'", log()->identify(cg->method()),
cg->unique_id());
JVMState* p = cg->call_node()->jvms();
while (p != nullptr) {
diff --git a/src/hotspot/share/opto/compile.hpp b/src/hotspot/share/opto/compile.hpp
index 9b6d8db05b0..69244a6d178 100644
--- a/src/hotspot/share/opto/compile.hpp
+++ b/src/hotspot/share/opto/compile.hpp
@@ -28,17 +28,18 @@
#include "asm/codeBuffer.hpp"
#include "ci/compilerInterface.hpp"
#include "code/debugInfoRec.hpp"
-#include "compiler/compiler_globals.hpp"
+#include "compiler/cHeapStringHolder.hpp"
#include "compiler/compileBroker.hpp"
+#include "compiler/compiler_globals.hpp"
#include "compiler/compilerEvent.hpp"
-#include "compiler/cHeapStringHolder.hpp"
#include "libadt/dict.hpp"
#include "libadt/vectset.hpp"
#include "memory/resourceArea.hpp"
#include "oops/methodData.hpp"
#include "opto/idealGraphPrinter.hpp"
-#include "opto/phasetype.hpp"
#include "opto/phase.hpp"
+#include "opto/phasetype.hpp"
+#include "opto/printinlining.hpp"
#include "opto/regmask.hpp"
#include "runtime/deoptimization.hpp"
#include "runtime/sharedRuntime.hpp"
@@ -46,7 +47,6 @@
#include "runtime/vmThread.hpp"
#include "utilities/ticks.hpp"
#include "utilities/vmEnums.hpp"
-#include "opto/printinlining.hpp"
class AbstractLockNode;
class AddPNode;
@@ -928,8 +928,8 @@ class Compile : public Phase {
bool copy_node_notes_to(Node* dest, Node* source);
// Workhorse function to sort out the blocked Node_Notes array:
- inline Node_Notes* locate_node_notes(GrowableArray* arr,
- int idx, bool can_grow = false);
+ Node_Notes* locate_node_notes(GrowableArray* arr,
+ int idx, bool can_grow = false);
void grow_node_notes(GrowableArray* arr, int grow_by);
diff --git a/src/hotspot/share/opto/convertnode.cpp b/src/hotspot/share/opto/convertnode.cpp
index c468c66fdfd..a495814da61 100644
--- a/src/hotspot/share/opto/convertnode.cpp
+++ b/src/hotspot/share/opto/convertnode.cpp
@@ -264,8 +264,68 @@ Node* ConvF2HFNode::Ideal(PhaseGVN* phase, bool can_reshape) {
return new ReinterpretHF2SNode(binop);
}
}
+
+ // Detects following ideal graph pattern
+ // ConvF2HF(binopF(conF, ConvHF2F(varS))) =>
+ // ReinterpretHF2SNode(binopHF(conHF, ReinterpretS2HFNode(varS)))
+ if (Float16NodeFactory::is_float32_binary_oper(in(1)->Opcode())) {
+ Node* binopF = in(1);
+ // Check if the incoming binary operation has one floating point constant
+ // input and the other input is a half precision to single precision upcasting node.
+ // We land here because a prior HalfFloat to Float conversion promotes
+ // an integral constant holding Float16 value to a floating point constant.
+ // i.e. ConvHF2F ConI(short) => ConF
+ Node* conF = nullptr;
+ Node* varS = nullptr;
+ if (binopF->in(1)->is_Con() && binopF->in(2)->Opcode() == Op_ConvHF2F) {
+ conF = binopF->in(1);
+ varS = binopF->in(2)->in(1);
+ } else if (binopF->in(2)->is_Con() && binopF->in(1)->Opcode() == Op_ConvHF2F) {
+ conF = binopF->in(2);
+ varS = binopF->in(1)->in(1);
+ }
+
+ if (conF != nullptr &&
+ varS != nullptr &&
+ conF->bottom_type()->isa_float_constant() != nullptr &&
+ Matcher::match_rule_supported(Float16NodeFactory::get_float16_binary_oper(binopF->Opcode())) &&
+ Matcher::match_rule_supported(Op_ReinterpretS2HF) &&
+ Matcher::match_rule_supported(Op_ReinterpretHF2S) &&
+ StubRoutines::hf2f_adr() != nullptr &&
+ StubRoutines::f2hf_adr() != nullptr) {
+ jfloat con = conF->bottom_type()->getf();
+ // Conditions under which floating point constant can be considered for a pattern match.
+ // 1. conF must lie within Float16 value range, otherwise we would have rounding issues:
+ // Doing the operation in float32 and then rounding is not the same as
+ // rounding first and doing the operation in float16.
+ // 2. If a constant value is one of the valid IEEE 754 binary32 NaN bit patterns
+ // then it's safe to consider it for pattern match because of the following reasons:
+ // a. As per section 2.8 of JVMS, Java Virtual Machine does not support
+ // signaling NaN value.
+ // b. Any signaling NaN which takes part in a non-comparison expression
+ // results in a quiet NaN but preserves the significand bits of signaling NaN.
+ // c. The pattern being matched includes a Float to Float16 conversion after binary
+ // expression, this downcast will still preserve the significand bits of binary32 NaN.
+ bool isnan = g_isnan((jdouble)con);
+ if (StubRoutines::hf2f(StubRoutines::f2hf(con)) == con || isnan) {
+ Node* newVarHF = phase->transform(new ReinterpretS2HFNode(varS));
+ Node* conHF = phase->makecon(TypeH::make(con));
+ Node* binopHF = nullptr;
+ // Preserving original input order for semantic correctness
+ // of non-commutative operation.
+ if (binopF->in(1) == conF) {
+ binopHF = phase->transform(Float16NodeFactory::make(binopF->Opcode(), binopF->in(0), conHF, newVarHF));
+ } else {
+ binopHF = phase->transform(Float16NodeFactory::make(binopF->Opcode(), binopF->in(0), newVarHF, conHF));
+ }
+ return new ReinterpretHF2SNode(binopHF);
+ }
+ }
+ }
+
return nullptr;
}
+
//=============================================================================
//------------------------------Value------------------------------------------
const Type* ConvF2INode::Value(PhaseGVN* phase) const {
diff --git a/src/hotspot/share/opto/divnode.cpp b/src/hotspot/share/opto/divnode.cpp
index a70194274a7..b3ac0a4a1b0 100644
--- a/src/hotspot/share/opto/divnode.cpp
+++ b/src/hotspot/share/opto/divnode.cpp
@@ -28,13 +28,13 @@
#include "opto/convertnode.hpp"
#include "opto/divnode.hpp"
#include "opto/machnode.hpp"
-#include "opto/movenode.hpp"
#include "opto/matcher.hpp"
+#include "opto/movenode.hpp"
#include "opto/mulnode.hpp"
#include "opto/phaseX.hpp"
+#include "opto/runtime.hpp"
#include "opto/subnode.hpp"
#include "utilities/powerOfTwo.hpp"
-#include "opto/runtime.hpp"
// Portions of code courtesy of Clifford Click
@@ -823,39 +823,11 @@ const Type* DivHFNode::Value(PhaseGVN* phase) const {
return bot;
}
- // x/x == 1, we ignore 0/0.
- // Note: if t1 and t2 are zero then result is NaN (JVMS page 213)
- // Does not work for variables because of NaN's
- if (in(1) == in(2) && t1->base() == Type::HalfFloatCon &&
- !g_isnan(t1->getf()) && g_isfinite(t1->getf()) && t1->getf() != 0.0) { // could be negative ZERO or NaN
- return TypeH::ONE;
- }
-
- if (t2 == TypeH::ONE) {
- return t1;
- }
-
- // If divisor is a constant and not zero, divide the numbers
if (t1->base() == Type::HalfFloatCon &&
- t2->base() == Type::HalfFloatCon &&
- t2->getf() != 0.0) {
- // could be negative zero
+ t2->base() == Type::HalfFloatCon) {
return TypeH::make(t1->getf() / t2->getf());
}
- // If the dividend is a constant zero
- // Note: if t1 and t2 are zero then result is NaN (JVMS page 213)
- // Test TypeHF::ZERO is not sufficient as it could be negative zero
-
- if (t1 == TypeH::ZERO && !g_isnan(t2->getf()) && t2->getf() != 0.0) {
- return TypeH::ZERO;
- }
-
- // If divisor or dividend is nan then result is nan.
- if (g_isnan(t1->getf()) || g_isnan(t2->getf())) {
- return TypeH::make(NAN);
- }
-
// Otherwise we give up all hope
return Type::HALF_FLOAT;
}
diff --git a/src/hotspot/share/opto/escape.cpp b/src/hotspot/share/opto/escape.cpp
index 3dd43f12b2f..295f7fca9f1 100644
--- a/src/hotspot/share/opto/escape.cpp
+++ b/src/hotspot/share/opto/escape.cpp
@@ -29,18 +29,18 @@
#include "libadt/vectset.hpp"
#include "memory/allocation.hpp"
#include "memory/resourceArea.hpp"
-#include "opto/c2compiler.hpp"
#include "opto/arraycopynode.hpp"
+#include "opto/c2compiler.hpp"
#include "opto/callnode.hpp"
+#include "opto/castnode.hpp"
#include "opto/cfgnode.hpp"
#include "opto/compile.hpp"
#include "opto/escape.hpp"
-#include "opto/macro.hpp"
#include "opto/locknode.hpp"
-#include "opto/phaseX.hpp"
+#include "opto/macro.hpp"
#include "opto/movenode.hpp"
#include "opto/narrowptrnode.hpp"
-#include "opto/castnode.hpp"
+#include "opto/phaseX.hpp"
#include "opto/rootnode.hpp"
#include "utilities/macros.hpp"
diff --git a/src/hotspot/share/opto/gcm.cpp b/src/hotspot/share/opto/gcm.cpp
index 6576196268e..8859263d8b6 100644
--- a/src/hotspot/share/opto/gcm.cpp
+++ b/src/hotspot/share/opto/gcm.cpp
@@ -29,12 +29,12 @@
#include "opto/c2compiler.hpp"
#include "opto/callnode.hpp"
#include "opto/cfgnode.hpp"
+#include "opto/chaitin.hpp"
#include "opto/machnode.hpp"
#include "opto/opcodes.hpp"
#include "opto/phaseX.hpp"
#include "opto/rootnode.hpp"
#include "opto/runtime.hpp"
-#include "opto/chaitin.hpp"
#include "runtime/deoptimization.hpp"
// Portions of code courtesy of Clifford Click
diff --git a/src/hotspot/share/opto/graphKit.cpp b/src/hotspot/share/opto/graphKit.cpp
index 20feca26ede..02d61596d31 100644
--- a/src/hotspot/share/opto/graphKit.cpp
+++ b/src/hotspot/share/opto/graphKit.cpp
@@ -22,10 +22,10 @@
*
*/
+#include "asm/register.hpp"
+#include "ci/ciObjArray.hpp"
#include "ci/ciUtilities.hpp"
#include "classfile/javaClasses.hpp"
-#include "ci/ciObjArray.hpp"
-#include "asm/register.hpp"
#include "compiler/compileLog.hpp"
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/c2/barrierSetC2.hpp"
@@ -47,8 +47,8 @@
#include "runtime/deoptimization.hpp"
#include "runtime/sharedRuntime.hpp"
#include "utilities/bitMap.inline.hpp"
-#include "utilities/powerOfTwo.hpp"
#include "utilities/growableArray.hpp"
+#include "utilities/powerOfTwo.hpp"
//----------------------------GraphKit-----------------------------------------
// Main utility constructor.
@@ -3803,6 +3803,8 @@ Node* GraphKit::new_array(Node* klass_node, // array klass (maybe variable)
assert(!StressReflectiveCode, "stress mode does not use these paths");
// Increase the size limit if we have exact knowledge of array type.
int log2_esize = Klass::layout_helper_log2_element_size(layout_con);
+ assert(fast_size_limit == 0 || count_leading_zeros(fast_size_limit) > static_cast(LogBytesPerLong - log2_esize),
+ "fast_size_limit (%d) overflow when shifted left by %d", fast_size_limit, LogBytesPerLong - log2_esize);
fast_size_limit <<= (LogBytesPerLong - log2_esize);
}
@@ -3853,7 +3855,9 @@ Node* GraphKit::new_array(Node* klass_node, // array klass (maybe variable)
if (tilen != nullptr && tilen->_lo < 0) {
// Add a manual constraint to a positive range. Cf. array_element_address.
jint size_max = fast_size_limit;
- if (size_max > tilen->_hi) size_max = tilen->_hi;
+ if (size_max > tilen->_hi && tilen->_hi >= 0) {
+ size_max = tilen->_hi;
+ }
const TypeInt* tlcon = TypeInt::make(0, size_max, Type::WidenMin);
// Only do a narrow I2L conversion if the range check passed.
diff --git a/src/hotspot/share/opto/idealGraphPrinter.cpp b/src/hotspot/share/opto/idealGraphPrinter.cpp
index 3721394c112..1ecb46eaf5a 100644
--- a/src/hotspot/share/opto/idealGraphPrinter.cpp
+++ b/src/hotspot/share/opto/idealGraphPrinter.cpp
@@ -155,7 +155,6 @@ void IdealGraphPrinter::init(const char* file_name, bool use_multiple_files, boo
// in the mach where kill projections have no users but should
// appear in the dump.
_traverse_outs = true;
- _should_send_method = true;
_output = nullptr;
buffer[0] = 0;
_depth = 0;
@@ -300,13 +299,11 @@ void IdealGraphPrinter::print_inline_tree(InlineTree *tree) {
void IdealGraphPrinter::print_inlining() {
// Print inline tree
- if (_should_send_method) {
- InlineTree *inlineTree = C->ilt();
- if (inlineTree != nullptr) {
- print_inline_tree(inlineTree);
- } else {
- // print this method only
- }
+ InlineTree *inlineTree = C->ilt();
+ if (inlineTree != nullptr) {
+ print_inline_tree(inlineTree);
+ } else {
+ // print this method only
}
}
@@ -382,7 +379,6 @@ void IdealGraphPrinter::begin_method() {
tail(PROPERTIES_ELEMENT);
- _should_send_method = true;
this->_current_method = method;
_xml->flush();
@@ -975,7 +971,7 @@ void IdealGraphPrinter::print_graph(const char* name, const frame* fr) {
// Print current ideal graph
void IdealGraphPrinter::print(const char* name, Node* node, GrowableArray& visible_nodes, const frame* fr) {
- if (!_current_method || !_should_send_method || node == nullptr) return;
+ if (!_current_method || node == nullptr) return;
if (name == nullptr) {
stringStream graph_name;
diff --git a/src/hotspot/share/opto/idealGraphPrinter.hpp b/src/hotspot/share/opto/idealGraphPrinter.hpp
index 69ba2841506..7e68ce6c00f 100644
--- a/src/hotspot/share/opto/idealGraphPrinter.hpp
+++ b/src/hotspot/share/opto/idealGraphPrinter.hpp
@@ -110,7 +110,6 @@ class IdealGraphPrinter : public CHeapObj {
ciMethod *_current_method;
int _depth;
char buffer[2048];
- bool _should_send_method;
PhaseChaitin* _chaitin;
bool _traverse_outs;
Compile *C;
diff --git a/src/hotspot/share/opto/idealKit.hpp b/src/hotspot/share/opto/idealKit.hpp
index 9916701fad7..fe07716dcff 100644
--- a/src/hotspot/share/opto/idealKit.hpp
+++ b/src/hotspot/share/opto/idealKit.hpp
@@ -26,8 +26,8 @@
#define SHARE_OPTO_IDEALKIT_HPP
#include "opto/addnode.hpp"
-#include "opto/cfgnode.hpp"
#include "opto/castnode.hpp"
+#include "opto/cfgnode.hpp"
#include "opto/connode.hpp"
#include "opto/divnode.hpp"
#include "opto/graphKit.hpp"
diff --git a/src/hotspot/share/opto/ifnode.cpp b/src/hotspot/share/opto/ifnode.cpp
index 8d810e4202f..9daf2e6741e 100644
--- a/src/hotspot/share/opto/ifnode.cpp
+++ b/src/hotspot/share/opto/ifnode.cpp
@@ -32,8 +32,8 @@
#include "opto/loopnode.hpp"
#include "opto/phaseX.hpp"
#include "opto/predicates_enums.hpp"
-#include "opto/runtime.hpp"
#include "opto/rootnode.hpp"
+#include "opto/runtime.hpp"
#include "opto/subnode.hpp"
#include "opto/subtypenode.hpp"
@@ -1046,8 +1046,7 @@ bool IfNode::fold_compares_helper(ProjNode* proj, ProjNode* success, ProjNode* f
if (failtype != nullptr) {
const TypeInt* type2 = filtered_int_type(igvn, n, fail);
if (type2 != nullptr) {
- failtype = failtype->join(type2)->is_int();
- if (failtype->empty()) {
+ if (failtype->filter(type2) == Type::TOP) {
// previous if determines the result of this if so
// replace Bool with constant
igvn->replace_input_of(this, 1, igvn->intcon(success->_con));
diff --git a/src/hotspot/share/opto/intrinsicnode.cpp b/src/hotspot/share/opto/intrinsicnode.cpp
index e67352d85bd..4f131a39f38 100644
--- a/src/hotspot/share/opto/intrinsicnode.cpp
+++ b/src/hotspot/share/opto/intrinsicnode.cpp
@@ -22,14 +22,14 @@
*
*/
-#include "opto/intrinsicnode.hpp"
#include "opto/addnode.hpp"
-#include "opto/mulnode.hpp"
+#include "opto/intrinsicnode.hpp"
#include "opto/memnode.hpp"
+#include "opto/mulnode.hpp"
#include "opto/phaseX.hpp"
-#include "utilities/population_count.hpp"
#include "utilities/count_leading_zeros.hpp"
#include "utilities/globalDefinitions.hpp"
+#include "utilities/population_count.hpp"
//=============================================================================
// Do not match memory edge.
@@ -135,7 +135,7 @@ Node* CompressBitsNode::Ideal(PhaseGVN* phase, bool can_reshape) {
Node* src = in(1);
Node* mask = in(2);
if (bottom_type()->isa_int()) {
- if (mask->Opcode() == Op_LShiftI && phase->type(mask->in(1))->is_int()->is_con()) {
+ if (mask->Opcode() == Op_LShiftI && phase->type(mask->in(1))->isa_int() && phase->type(mask->in(1))->is_int()->is_con()) {
// compress(x, 1 << n) == (x >> n & 1)
if (phase->type(mask->in(1))->higher_equal(TypeInt::ONE)) {
Node* rshift = phase->transform(new RShiftINode(in(1), mask->in(2)));
@@ -153,7 +153,7 @@ Node* CompressBitsNode::Ideal(PhaseGVN* phase, bool can_reshape) {
}
} else {
assert(bottom_type()->isa_long(), "");
- if (mask->Opcode() == Op_LShiftL && phase->type(mask->in(1))->is_long()->is_con()) {
+ if (mask->Opcode() == Op_LShiftL && phase->type(mask->in(1))->isa_long() && phase->type(mask->in(1))->is_long()->is_con()) {
// compress(x, 1 << n) == (x >> n & 1)
if (phase->type(mask->in(1))->higher_equal(TypeLong::ONE)) {
Node* rshift = phase->transform(new RShiftLNode(in(1), mask->in(2)));
@@ -193,7 +193,7 @@ Node* ExpandBitsNode::Ideal(PhaseGVN* phase, bool can_reshape) {
Node* src = in(1);
Node* mask = in(2);
if (bottom_type()->isa_int()) {
- if (mask->Opcode() == Op_LShiftI && phase->type(mask->in(1))->is_int()->is_con()) {
+ if (mask->Opcode() == Op_LShiftI && phase->type(mask->in(1))->isa_int() && phase->type(mask->in(1))->is_int()->is_con()) {
// expand(x, 1 << n) == (x & 1) << n
if (phase->type(mask->in(1))->higher_equal(TypeInt::ONE)) {
Node* andnode = phase->transform(new AndINode(in(1), phase->makecon(TypeInt::ONE)));
@@ -210,7 +210,7 @@ Node* ExpandBitsNode::Ideal(PhaseGVN* phase, bool can_reshape) {
}
} else {
assert(bottom_type()->isa_long(), "");
- if (mask->Opcode() == Op_LShiftL && phase->type(mask->in(1))->is_long()->is_con()) {
+ if (mask->Opcode() == Op_LShiftL && phase->type(mask->in(1))->isa_long() && phase->type(mask->in(1))->is_long()->is_con()) {
// expand(x, 1 << n) == (x & 1) << n
if (phase->type(mask->in(1))->higher_equal(TypeLong::ONE)) {
Node* andnode = phase->transform(new AndLNode(in(1), phase->makecon(TypeLong::ONE)));
diff --git a/src/hotspot/share/opto/intrinsicnode.hpp b/src/hotspot/share/opto/intrinsicnode.hpp
index 3c204f88bc6..df34e363259 100644
--- a/src/hotspot/share/opto/intrinsicnode.hpp
+++ b/src/hotspot/share/opto/intrinsicnode.hpp
@@ -25,9 +25,9 @@
#ifndef SHARE_OPTO_INTRINSICNODE_HPP
#define SHARE_OPTO_INTRINSICNODE_HPP
+#include "opto/connode.hpp"
#include "opto/node.hpp"
#include "opto/opcodes.hpp"
-#include "opto/connode.hpp"
//----------------------PartialSubtypeCheckNode--------------------------------
diff --git a/src/hotspot/share/opto/lcm.cpp b/src/hotspot/share/opto/lcm.cpp
index 0fe246a7679..66ef4b23e2d 100644
--- a/src/hotspot/share/opto/lcm.cpp
+++ b/src/hotspot/share/opto/lcm.cpp
@@ -31,9 +31,9 @@
#include "opto/c2compiler.hpp"
#include "opto/callnode.hpp"
#include "opto/cfgnode.hpp"
+#include "opto/chaitin.hpp"
#include "opto/machnode.hpp"
#include "opto/runtime.hpp"
-#include "opto/chaitin.hpp"
#include "runtime/os.inline.hpp"
#include "runtime/sharedRuntime.hpp"
diff --git a/src/hotspot/share/opto/library_call.cpp b/src/hotspot/share/opto/library_call.cpp
index 29f737bce08..94f047dc115 100644
--- a/src/hotspot/share/opto/library_call.cpp
+++ b/src/hotspot/share/opto/library_call.cpp
@@ -23,8 +23,8 @@
*/
#include "asm/macroAssembler.hpp"
-#include "ci/ciUtilities.inline.hpp"
#include "ci/ciSymbols.hpp"
+#include "ci/ciUtilities.inline.hpp"
#include "classfile/vmIntrinsics.hpp"
#include "compiler/compileBroker.hpp"
#include "compiler/compileLog.hpp"
@@ -47,8 +47,8 @@
#include "opto/narrowptrnode.hpp"
#include "opto/opaquenode.hpp"
#include "opto/parse.hpp"
-#include "opto/runtime.hpp"
#include "opto/rootnode.hpp"
+#include "opto/runtime.hpp"
#include "opto/subnode.hpp"
#include "opto/vectornode.hpp"
#include "prims/jvmtiExport.hpp"
@@ -564,7 +564,7 @@ bool LibraryCallKit::try_to_inline(int predicate) {
case vmIntrinsics::_getCallerClass: return inline_native_Reflection_getCallerClass();
- case vmIntrinsics::_Reference_get: return inline_reference_get();
+ case vmIntrinsics::_Reference_get0: return inline_reference_get0();
case vmIntrinsics::_Reference_refersTo0: return inline_reference_refersTo0(false);
case vmIntrinsics::_PhantomReference_refersTo0: return inline_reference_refersTo0(true);
case vmIntrinsics::_Reference_clear0: return inline_reference_clear0(false);
@@ -1575,9 +1575,14 @@ bool LibraryCallKit::inline_string_toBytesU() {
Node* src_start = array_element_address(value, offset, T_CHAR);
Node* dst_start = basic_plus_adr(newcopy, arrayOopDesc::base_offset_in_bytes(T_BYTE));
- // Check if src array address is aligned to HeapWordSize (dst is always aligned)
- const TypeInt* toffset = gvn().type(offset)->is_int();
- bool aligned = toffset->is_con() && ((toffset->get_con() * type2aelembytes(T_CHAR)) % HeapWordSize == 0);
+ // Check if dst array address is aligned to HeapWordSize
+ bool aligned = (arrayOopDesc::base_offset_in_bytes(T_BYTE) % HeapWordSize == 0);
+ // If true, then check if src array address is aligned to HeapWordSize
+ if (aligned) {
+ const TypeInt* toffset = gvn().type(offset)->is_int();
+ aligned = toffset->is_con() && ((arrayOopDesc::base_offset_in_bytes(T_CHAR) +
+ toffset->get_con() * type2aelembytes(T_CHAR)) % HeapWordSize == 0);
+ }
// Figure out which arraycopy runtime method to call (disjoint, uninitialized).
const char* copyfunc_name = "arraycopy";
@@ -1658,8 +1663,8 @@ bool LibraryCallKit::inline_string_getCharsU() {
// Check if array addresses are aligned to HeapWordSize
const TypeInt* tsrc = gvn().type(src_begin)->is_int();
const TypeInt* tdst = gvn().type(dst_begin)->is_int();
- bool aligned = tsrc->is_con() && ((tsrc->get_con() * type2aelembytes(T_BYTE)) % HeapWordSize == 0) &&
- tdst->is_con() && ((tdst->get_con() * type2aelembytes(T_CHAR)) % HeapWordSize == 0);
+ bool aligned = tsrc->is_con() && ((arrayOopDesc::base_offset_in_bytes(T_BYTE) + tsrc->get_con() * type2aelembytes(T_BYTE)) % HeapWordSize == 0) &&
+ tdst->is_con() && ((arrayOopDesc::base_offset_in_bytes(T_CHAR) + tdst->get_con() * type2aelembytes(T_CHAR)) % HeapWordSize == 0);
// Figure out which arraycopy runtime method to call (disjoint, uninitialized).
const char* copyfunc_name = "arraycopy";
@@ -6580,6 +6585,10 @@ bool LibraryCallKit::inline_vectorizedMismatch() {
memory_phi = _gvn.transform(memory_phi);
result_phi = _gvn.transform(result_phi);
+ record_for_igvn(exit_block);
+ record_for_igvn(memory_phi);
+ record_for_igvn(result_phi);
+
set_control(exit_block);
set_all_memory(memory_phi);
set_result(result_phi);
@@ -6914,9 +6923,9 @@ bool LibraryCallKit::inline_updateByteBufferAdler32() {
return true;
}
-//----------------------------inline_reference_get----------------------------
+//----------------------------inline_reference_get0----------------------------
// public T java.lang.ref.Reference.get();
-bool LibraryCallKit::inline_reference_get() {
+bool LibraryCallKit::inline_reference_get0() {
const int referent_offset = java_lang_ref_Reference::referent_offset();
// Get the argument:
diff --git a/src/hotspot/share/opto/library_call.hpp b/src/hotspot/share/opto/library_call.hpp
index 1be08df32ae..94b02290ccd 100644
--- a/src/hotspot/share/opto/library_call.hpp
+++ b/src/hotspot/share/opto/library_call.hpp
@@ -22,12 +22,15 @@
*
*/
+#ifndef SHARE_OPTO_LIBRARY_CALL_HPP
+#define SHARE_OPTO_LIBRARY_CALL_HPP
+
#include "ci/ciMethod.hpp"
#include "classfile/javaClasses.hpp"
#include "opto/callGenerator.hpp"
-#include "opto/graphKit.hpp"
#include "opto/castnode.hpp"
#include "opto/convertnode.hpp"
+#include "opto/graphKit.hpp"
#include "opto/intrinsicnode.hpp"
#include "opto/movenode.hpp"
@@ -300,7 +303,7 @@ class LibraryCallKit : public GraphKit {
bool inline_bitshuffle_methods(vmIntrinsics::ID id);
bool inline_compare_unsigned(vmIntrinsics::ID id);
bool inline_divmod_methods(vmIntrinsics::ID id);
- bool inline_reference_get();
+ bool inline_reference_get0();
bool inline_reference_refersTo0(bool is_phantom);
bool inline_reference_clear0(bool is_phantom);
bool inline_Class_cast();
@@ -416,3 +419,4 @@ class LibraryCallKit : public GraphKit {
bool inline_blackhole();
};
+#endif // SHARE_OPTO_LIBRARY_CALL_HPP
diff --git a/src/hotspot/share/opto/loopPredicate.cpp b/src/hotspot/share/opto/loopPredicate.cpp
index 96e0fd26d0d..477ea48d419 100644
--- a/src/hotspot/share/opto/loopPredicate.cpp
+++ b/src/hotspot/share/opto/loopPredicate.cpp
@@ -35,6 +35,7 @@
#include "opto/predicates.hpp"
#include "opto/rootnode.hpp"
#include "opto/subnode.hpp"
+
#include
#include
diff --git a/src/hotspot/share/opto/loopTransform.cpp b/src/hotspot/share/opto/loopTransform.cpp
index 58105afd9d5..afe6a076268 100644
--- a/src/hotspot/share/opto/loopTransform.cpp
+++ b/src/hotspot/share/opto/loopTransform.cpp
@@ -33,8 +33,8 @@
#include "opto/convertnode.hpp"
#include "opto/divnode.hpp"
#include "opto/loopnode.hpp"
-#include "opto/mulnode.hpp"
#include "opto/movenode.hpp"
+#include "opto/mulnode.hpp"
#include "opto/opaquenode.hpp"
#include "opto/phase.hpp"
#include "opto/predicates.hpp"
diff --git a/src/hotspot/share/opto/loopnode.cpp b/src/hotspot/share/opto/loopnode.cpp
index 3130df54681..e5efdb2a202 100644
--- a/src/hotspot/share/opto/loopnode.cpp
+++ b/src/hotspot/share/opto/loopnode.cpp
@@ -298,19 +298,52 @@ IdealLoopTree* PhaseIdealLoop::insert_outer_loop(IdealLoopTree* loop, LoopNode*
return outer_ilt;
}
-// Create a skeleton strip mined outer loop: a Loop head before the
-// inner strip mined loop, a safepoint and an exit condition guarded
-// by an opaque node after the inner strip mined loop with a backedge
-// to the loop head. The inner strip mined loop is left as it is. Only
-// once loop optimizations are over, do we adjust the inner loop exit
-// condition to limit its number of iterations, set the outer loop
-// exit condition and add Phis to the outer loop head. Some loop
-// optimizations that operate on the inner strip mined loop need to be
-// aware of the outer strip mined loop: loop unswitching needs to
-// clone the outer loop as well as the inner, unrolling needs to only
-// clone the inner loop etc. No optimizations need to change the outer
-// strip mined loop as it is only a skeleton.
-IdealLoopTree* PhaseIdealLoop::create_outer_strip_mined_loop(BoolNode *test, Node *cmp, Node *init_control,
+// Create a skeleton strip mined outer loop: an OuterStripMinedLoop head before the inner strip mined CountedLoop, a
+// SafePoint on exit of the inner CountedLoopEnd and an OuterStripMinedLoopEnd test that can't constant fold until loop
+// optimizations are over. The inner strip mined loop is left as it is. Only once loop optimizations are over, do we
+// adjust the inner loop exit condition to limit its number of iterations, set the outer loop exit condition and add
+// Phis to the outer loop head. Some loop optimizations that operate on the inner strip mined loop need to be aware of
+// the outer strip mined loop: loop unswitching needs to clone the outer loop as well as the inner, unrolling needs to
+// only clone the inner loop etc. No optimizations need to change the outer strip mined loop as it is only a skeleton.
+//
+// Schematically:
+//
+// OuterStripMinedLoop -------|
+// | |
+// CountedLoop ----------- | |
+// \- Phi (iv) -| | |
+// / \ | | |
+// CmpI AddI --| | |
+// \ | |
+// Bool | |
+// \ | |
+// CountedLoopEnd | |
+// / \ | |
+// IfFalse IfTrue--------| |
+// | |
+// SafePoint |
+// | |
+// OuterStripMinedLoopEnd |
+// / \ |
+// IfFalse IfTrue-----------|
+// |
+//
+//
+// As loop optimizations transform the inner loop, the outer strip mined loop stays mostly unchanged. The only exception
+// is nodes referenced from the SafePoint and sunk from the inner loop: they end up in the outer strip mined loop.
+//
+// Not adding Phis to the outer loop head from the beginning, and only adding them after loop optimizations does not
+// conform to C2's IR rules: any variable or memory slice that is mutated in a loop should have a Phi. The main
+// motivation for such a design that doesn't conform to C2's IR rules is to allow existing loop optimizations to be
+// mostly unaffected by the outer strip mined loop: the only extra step needed in most cases is to step over the
+// OuterStripMinedLoop. The main drawback is that once loop optimizations are over, an extra step is needed to finish
+// constructing the outer loop. This is handled by OuterStripMinedLoopNode::adjust_strip_mined_loop().
+//
+// Adding Phis to the outer loop is largely straightforward: there needs to be one Phi in the outer loop for every Phi
+// in the inner loop. Things may be more complicated for sunk Store nodes: there may not be any inner loop Phi left
+// after sinking for a particular memory slice but the outer loop needs a Phi. See
+// OuterStripMinedLoopNode::handle_sunk_stores_when_finishing_construction()
+IdealLoopTree* PhaseIdealLoop::create_outer_strip_mined_loop(Node* init_control,
IdealLoopTree* loop, float cl_prob, float le_fcnt,
Node*& entry_control, Node*& iffalse) {
Node* outer_test = intcon(0);
@@ -1656,7 +1689,9 @@ bool PhaseIdealLoop::is_counted_loop(Node* x, IdealLoopTree*&loop, BasicType iv_
return false;
}
- if (xphi->Opcode() == Op_Cast(iv_bt)) {
+ // Iteratively uncast the loop induction variable
+ // until no more CastII/CastLL nodes are found.
+ while (xphi->Opcode() == Op_Cast(iv_bt)) {
xphi = xphi->in(1);
}
@@ -2255,9 +2290,8 @@ bool PhaseIdealLoop::is_counted_loop(Node* x, IdealLoopTree*&loop, BasicType iv_
is_deleteable_safept(sfpt);
IdealLoopTree* outer_ilt = nullptr;
if (strip_mine_loop) {
- outer_ilt = create_outer_strip_mined_loop(test, cmp, init_control, loop,
- cl_prob, le->_fcnt, entry_control,
- iffalse);
+ outer_ilt = create_outer_strip_mined_loop(init_control, loop, cl_prob, le->_fcnt,
+ entry_control, iffalse);
}
// Now setup a new CountedLoopNode to replace the existing LoopNode
@@ -2870,10 +2904,11 @@ BaseCountedLoopNode* BaseCountedLoopNode::make(Node* entry, Node* backedge, Basi
return new LongCountedLoopNode(entry, backedge);
}
-void OuterStripMinedLoopNode::fix_sunk_stores(CountedLoopEndNode* inner_cle, LoopNode* inner_cl, PhaseIterGVN* igvn,
- PhaseIdealLoop* iloop) {
- Node* cle_out = inner_cle->proj_out(false);
- Node* cle_tail = inner_cle->proj_out(true);
+void OuterStripMinedLoopNode::fix_sunk_stores_when_back_to_counted_loop(PhaseIterGVN* igvn,
+ PhaseIdealLoop* iloop) const {
+ CountedLoopNode* inner_cl = inner_counted_loop();
+ IfFalseNode* cle_out = inner_loop_exit();
+
if (cle_out->outcnt() > 1) {
// Look for chains of stores that were sunk
// out of the inner loop and are in the outer loop
@@ -2988,11 +3023,90 @@ void OuterStripMinedLoopNode::fix_sunk_stores(CountedLoopEndNode* inner_cle, Loo
}
}
+// The outer strip mined loop is initially only partially constructed. In particular Phis are omitted.
+// See comment above: PhaseIdealLoop::create_outer_strip_mined_loop()
+// We're now in the process of finishing the construction of the outer loop. For each Phi in the inner loop, a Phi in
+// the outer loop was just now created. However, Sunk Stores cause an extra challenge:
+// 1) If all Stores in the inner loop were sunk for a particular memory slice, there's no Phi left for that memory slice
+// in the inner loop anymore, and hence we did not yet add a Phi for the outer loop. So an extra Phi must now be
+// added for each chain of sunk Stores for a particular memory slice.
+// 2) If some Stores were sunk and some left in the inner loop, a Phi was already created in the outer loop but
+// its backedge input wasn't wired correctly to the last Store of the chain: the backedge input was set to the
+// backedge of the inner loop Phi instead, but it needs to be the last Store of the chain in the outer loop. We now
+// have to fix that too.
+void OuterStripMinedLoopNode::handle_sunk_stores_when_finishing_construction(PhaseIterGVN* igvn) {
+ IfFalseNode* cle_exit_proj = inner_loop_exit();
+
+ // Find Sunk stores: Sunk stores are pinned on the loop exit projection of the inner loop. Indeed, because Sunk Stores
+ // modify the memory state captured by the SafePoint in the outer strip mined loop, they must be above it. The
+ // SafePoint's control input is the loop exit projection. It's also the only control out of the inner loop above the
+ // SafePoint.
+#ifdef ASSERT
+ int stores_in_outer_loop_cnt = 0;
+ for (DUIterator_Fast imax, i = cle_exit_proj->fast_outs(imax); i < imax; i++) {
+ Node* u = cle_exit_proj->fast_out(i);
+ if (u->is_Store()) {
+ stores_in_outer_loop_cnt++;
+ }
+ }
+#endif
+
+ // Sunk stores are reachable from the memory state of the outer loop safepoint
+ SafePointNode* safepoint = outer_safepoint();
+ MergeMemNode* mm = safepoint->in(TypeFunc::Memory)->isa_MergeMem();
+ if (mm == nullptr) {
+ // There is no MergeMem, which should only happen if there was no memory node
+ // sunk out of the loop.
+ assert(stores_in_outer_loop_cnt == 0, "inconsistent");
+ return;
+ }
+ DEBUG_ONLY(int stores_in_outer_loop_cnt2 = 0);
+ for (MergeMemStream mms(mm); mms.next_non_empty();) {
+ Node* mem = mms.memory();
+ // Traverse up the chain of stores to find the first store pinned
+ // at the loop exit projection.
+ Node* last = mem;
+ Node* first = nullptr;
+ while (mem->is_Store() && mem->in(0) == cle_exit_proj) {
+ DEBUG_ONLY(stores_in_outer_loop_cnt2++);
+ first = mem;
+ mem = mem->in(MemNode::Memory);
+ }
+ if (first != nullptr) {
+ // Found a chain of Stores that were sunk
+ // Do we already have a memory Phi for that slice on the outer loop? If that is the case, that Phi was created
+ // by cloning an inner loop Phi. The inner loop Phi should have mem, the memory state of the first Store out of
+ // the inner loop, as input on the backedge. So does the outer loop Phi given it's a clone.
+ Node* phi = nullptr;
+ for (DUIterator_Fast imax, i = mem->fast_outs(imax); i < imax; i++) {
+ Node* u = mem->fast_out(i);
+ if (u->is_Phi() && u->in(0) == this && u->in(LoopBackControl) == mem) {
+ assert(phi == nullptr, "there should be only one");
+ phi = u;
+ PRODUCT_ONLY(break);
+ }
+ }
+ if (phi == nullptr) {
+ // No outer loop Phi? create one
+ phi = PhiNode::make(this, last);
+ phi->set_req(EntryControl, mem);
+ phi = igvn->transform(phi);
+ igvn->replace_input_of(first, MemNode::Memory, phi);
+ } else {
+ // Fix memory state along the backedge: it should be the last sunk Store of the chain
+ igvn->replace_input_of(phi, LoopBackControl, last);
+ }
+ }
+ }
+ assert(stores_in_outer_loop_cnt == stores_in_outer_loop_cnt2, "inconsistent");
+}
+
void OuterStripMinedLoopNode::adjust_strip_mined_loop(PhaseIterGVN* igvn) {
+ verify_strip_mined(1);
// Look for the outer & inner strip mined loop, reduce number of
// iterations of the inner loop, set exit condition of outer loop,
// construct required phi nodes for outer loop.
- CountedLoopNode* inner_cl = unique_ctrl_out()->as_CountedLoop();
+ CountedLoopNode* inner_cl = inner_counted_loop();
assert(inner_cl->is_strip_mined(), "inner loop should be strip mined");
if (LoopStripMiningIter == 0) {
remove_outer_loop_and_safepoint(igvn);
@@ -3010,7 +3124,7 @@ void OuterStripMinedLoopNode::adjust_strip_mined_loop(PhaseIterGVN* igvn) {
inner_cl->clear_strip_mined();
return;
}
- CountedLoopEndNode* inner_cle = inner_cl->loopexit();
+ CountedLoopEndNode* inner_cle = inner_counted_loop_end();
int stride = inner_cl->stride_con();
// For a min int stride, LoopStripMiningIter * stride overflows the int range for all values of LoopStripMiningIter
@@ -3091,8 +3205,9 @@ void OuterStripMinedLoopNode::adjust_strip_mined_loop(PhaseIterGVN* igvn) {
}
Node* iv_phi = nullptr;
- // Make a clone of each phi in the inner loop
- // for the outer loop
+ // Make a clone of each phi in the inner loop for the outer loop
+ // When Stores were Sunk, after this step, a Phi may still be missing or its backedge incorrectly wired. See
+ // handle_sunk_stores_when_finishing_construction()
for (uint i = 0; i < inner_cl->outcnt(); i++) {
Node* u = inner_cl->raw_out(i);
if (u->is_Phi()) {
@@ -3111,6 +3226,8 @@ void OuterStripMinedLoopNode::adjust_strip_mined_loop(PhaseIterGVN* igvn) {
}
}
+ handle_sunk_stores_when_finishing_construction(igvn);
+
if (iv_phi != nullptr) {
// Now adjust the inner loop's exit condition
Node* limit = inner_cl->limit();
@@ -3166,7 +3283,7 @@ void OuterStripMinedLoopNode::transform_to_counted_loop(PhaseIterGVN* igvn, Phas
CountedLoopEndNode* inner_cle = inner_cl->loopexit();
Node* safepoint = outer_safepoint();
- fix_sunk_stores(inner_cle, inner_cl, igvn, iloop);
+ fix_sunk_stores_when_back_to_counted_loop(igvn, iloop);
// make counted loop exit test always fail
ConINode* zero = igvn->intcon(0);
diff --git a/src/hotspot/share/opto/loopnode.hpp b/src/hotspot/share/opto/loopnode.hpp
index 7670f6c620e..e056bda28e2 100644
--- a/src/hotspot/share/opto/loopnode.hpp
+++ b/src/hotspot/share/opto/loopnode.hpp
@@ -573,7 +573,8 @@ class LoopLimitNode : public Node {
// Support for strip mining
class OuterStripMinedLoopNode : public LoopNode {
private:
- static void fix_sunk_stores(CountedLoopEndNode* inner_cle, LoopNode* inner_cl, PhaseIterGVN* igvn, PhaseIdealLoop* iloop);
+ void fix_sunk_stores_when_back_to_counted_loop(PhaseIterGVN* igvn, PhaseIdealLoop* iloop) const;
+ void handle_sunk_stores_when_finishing_construction(PhaseIterGVN* igvn);
public:
OuterStripMinedLoopNode(Compile* C, Node *entry, Node *backedge)
@@ -589,6 +590,10 @@ class OuterStripMinedLoopNode : public LoopNode {
virtual OuterStripMinedLoopEndNode* outer_loop_end() const;
virtual IfFalseNode* outer_loop_exit() const;
virtual SafePointNode* outer_safepoint() const;
+ CountedLoopNode* inner_counted_loop() const { return unique_ctrl_out()->as_CountedLoop(); }
+ CountedLoopEndNode* inner_counted_loop_end() const { return inner_counted_loop()->loopexit(); }
+ IfFalseNode* inner_loop_exit() const { return inner_counted_loop_end()->proj_out(false)->as_IfFalse(); }
+
void adjust_strip_mined_loop(PhaseIterGVN* igvn);
void remove_outer_loop_and_safepoint(PhaseIterGVN* igvn) const;
@@ -1293,7 +1298,7 @@ class PhaseIdealLoop : public PhaseTransform {
void add_parse_predicate(Deoptimization::DeoptReason reason, Node* inner_head, IdealLoopTree* loop, SafePointNode* sfpt);
SafePointNode* find_safepoint(Node* back_control, Node* x, IdealLoopTree* loop);
IdealLoopTree* insert_outer_loop(IdealLoopTree* loop, LoopNode* outer_l, Node* outer_ift);
- IdealLoopTree* create_outer_strip_mined_loop(BoolNode *test, Node *cmp, Node *init_control,
+ IdealLoopTree* create_outer_strip_mined_loop(Node* init_control,
IdealLoopTree* loop, float cl_prob, float le_fcnt,
Node*& entry_control, Node*& iffalse);
diff --git a/src/hotspot/share/opto/loopopts.cpp b/src/hotspot/share/opto/loopopts.cpp
index ddeb5f84d49..b33895adf5f 100644
--- a/src/hotspot/share/opto/loopopts.cpp
+++ b/src/hotspot/share/opto/loopopts.cpp
@@ -30,12 +30,11 @@
#include "opto/callnode.hpp"
#include "opto/castnode.hpp"
#include "opto/connode.hpp"
-#include "opto/castnode.hpp"
#include "opto/divnode.hpp"
#include "opto/loopnode.hpp"
#include "opto/matcher.hpp"
-#include "opto/mulnode.hpp"
#include "opto/movenode.hpp"
+#include "opto/mulnode.hpp"
#include "opto/opaquenode.hpp"
#include "opto/rootnode.hpp"
#include "opto/subnode.hpp"
diff --git a/src/hotspot/share/opto/macroArrayCopy.cpp b/src/hotspot/share/opto/macroArrayCopy.cpp
index e2209fddbdf..10de940c0c2 100644
--- a/src/hotspot/share/opto/macroArrayCopy.cpp
+++ b/src/hotspot/share/opto/macroArrayCopy.cpp
@@ -24,14 +24,14 @@
#include "gc/shared/barrierSet.hpp"
#include "gc/shared/tlab_globals.hpp"
-#include "opto/arraycopynode.hpp"
#include "oops/objArrayKlass.hpp"
+#include "opto/arraycopynode.hpp"
+#include "opto/castnode.hpp"
#include "opto/convertnode.hpp"
-#include "opto/vectornode.hpp"
#include "opto/graphKit.hpp"
#include "opto/macro.hpp"
#include "opto/runtime.hpp"
-#include "opto/castnode.hpp"
+#include "opto/vectornode.hpp"
#include "runtime/stubRoutines.hpp"
#include "utilities/align.hpp"
#include "utilities/powerOfTwo.hpp"
diff --git a/src/hotspot/share/opto/matcher.hpp b/src/hotspot/share/opto/matcher.hpp
index baf43b0d538..2ee2ded17b6 100644
--- a/src/hotspot/share/opto/matcher.hpp
+++ b/src/hotspot/share/opto/matcher.hpp
@@ -65,13 +65,8 @@ class Matcher : public PhaseTransform {
Node_Stack::push(n, (uint)ns);
}
void push(Node *n, Node_State ns, Node *parent, int indx) {
- ++_inode_top;
- if ((_inode_top + 1) >= _inode_max) grow();
- _inode_top->node = parent;
- _inode_top->indx = (uint)indx;
- ++_inode_top;
- _inode_top->node = n;
- _inode_top->indx = (uint)ns;
+ Node_Stack::push(parent, (uint)indx);
+ Node_Stack::push(n, (uint)ns);
}
Node *parent() {
pop();
diff --git a/src/hotspot/share/opto/memnode.cpp b/src/hotspot/share/opto/memnode.cpp
index 24b81b894cb..1da8f429359 100644
--- a/src/hotspot/share/opto/memnode.cpp
+++ b/src/hotspot/share/opto/memnode.cpp
@@ -34,7 +34,6 @@
#include "opto/addnode.hpp"
#include "opto/arraycopynode.hpp"
#include "opto/cfgnode.hpp"
-#include "opto/regalloc.hpp"
#include "opto/compile.hpp"
#include "opto/connode.hpp"
#include "opto/convertnode.hpp"
@@ -46,6 +45,7 @@
#include "opto/mulnode.hpp"
#include "opto/narrowptrnode.hpp"
#include "opto/phaseX.hpp"
+#include "opto/regalloc.hpp"
#include "opto/regmask.hpp"
#include "opto/rootnode.hpp"
#include "opto/traceMergeStoresTag.hpp"
diff --git a/src/hotspot/share/opto/mempointer.cpp b/src/hotspot/share/opto/mempointer.cpp
index bede753ff07..e400e40e4a2 100644
--- a/src/hotspot/share/opto/mempointer.cpp
+++ b/src/hotspot/share/opto/mempointer.cpp
@@ -22,10 +22,10 @@
*
*/
-#include "opto/mempointer.hpp"
+#include "classfile/vmSymbols.hpp"
#include "opto/addnode.hpp"
+#include "opto/mempointer.hpp"
#include "utilities/resourceHash.hpp"
-#include "classfile/vmSymbols.hpp"
MemPointerParserCallback MemPointerParserCallback::_empty;
diff --git a/src/hotspot/share/opto/node.cpp b/src/hotspot/share/opto/node.cpp
index 7633f68ea1e..8f6c67c16f5 100644
--- a/src/hotspot/share/opto/node.cpp
+++ b/src/hotspot/share/opto/node.cpp
@@ -2798,7 +2798,6 @@ const RegMask &Node::in_RegMask(uint) const {
}
void Node_Array::grow(uint i) {
- _nesting.check(_a); // Check if a potential reallocation in the arena is safe
assert(i >= _max, "Should have been checked before, use maybe_grow?");
assert(_max > 0, "invariant");
uint old = _max;
@@ -3038,10 +3037,6 @@ void Unique_Node_List::remove_useless_nodes(VectorSet &useful) {
//=============================================================================
void Node_Stack::grow() {
- _nesting.check(_a); // Check if a potential reallocation in the arena is safe
- if (_inode_top < _inode_max) {
- return; // No need to grow
- }
size_t old_top = pointer_delta(_inode_top,_inodes,sizeof(INode)); // save _top
size_t old_max = pointer_delta(_inode_max,_inodes,sizeof(INode));
size_t max = old_max << 1; // max * 2
diff --git a/src/hotspot/share/opto/node.hpp b/src/hotspot/share/opto/node.hpp
index 843baf48cf8..2bbb10879f5 100644
--- a/src/hotspot/share/opto/node.hpp
+++ b/src/hotspot/share/opto/node.hpp
@@ -1633,6 +1633,7 @@ class Node_Array : public AnyObj {
// Grow array to required capacity
void maybe_grow(uint i) {
+ _nesting.check(_a); // Check if a potential reallocation in the arena is safe
if (i >= _max) {
grow(i);
}
@@ -1884,7 +1885,15 @@ class Node_Stack {
INode *_inodes; // Array storage for the stack
Arena *_a; // Arena to allocate in
ReallocMark _nesting; // Safety checks for arena reallocation
+
+ void maybe_grow() {
+ _nesting.check(_a); // Check if a potential reallocation in the arena is safe
+ if (_inode_top >= _inode_max) {
+ grow();
+ }
+ }
void grow();
+
public:
Node_Stack(int size) {
size_t max = (size > OptoNodeListSize) ? size : OptoNodeListSize;
@@ -1907,7 +1916,7 @@ class Node_Stack {
}
void push(Node *n, uint i) {
++_inode_top;
- grow();
+ maybe_grow();
INode *top = _inode_top; // optimization
top->node = n;
top->indx = i;
diff --git a/src/hotspot/share/opto/output.cpp b/src/hotspot/share/opto/output.cpp
index f0a7b742998..cadc2511770 100644
--- a/src/hotspot/share/opto/output.cpp
+++ b/src/hotspot/share/opto/output.cpp
@@ -36,8 +36,8 @@
#include "memory/allocation.hpp"
#include "opto/ad.hpp"
#include "opto/block.hpp"
-#include "opto/c2compiler.hpp"
#include "opto/c2_MacroAssembler.hpp"
+#include "opto/c2compiler.hpp"
#include "opto/callnode.hpp"
#include "opto/cfgnode.hpp"
#include "opto/locknode.hpp"
diff --git a/src/hotspot/share/opto/phaseX.cpp b/src/hotspot/share/opto/phaseX.cpp
index 953afb0e830..1ba8e145e7d 100644
--- a/src/hotspot/share/opto/phaseX.cpp
+++ b/src/hotspot/share/opto/phaseX.cpp
@@ -1072,7 +1072,11 @@ void PhaseIterGVN::optimize() {
#ifdef ASSERT
void PhaseIterGVN::verify_optimize() {
- if (is_verify_Value()) {
+ assert(_worklist.size() == 0, "igvn worklist must be empty before verify");
+
+ if (is_verify_Value() ||
+ is_verify_Ideal() ||
+ is_verify_Identity()) {
ResourceMark rm;
Unique_Node_List worklist;
bool failure = false;
@@ -1080,7 +1084,10 @@ void PhaseIterGVN::verify_optimize() {
worklist.push(C->root());
for (uint j = 0; j < worklist.size(); ++j) {
Node* n = worklist.at(j);
- failure |= verify_node_value(n);
+ if (is_verify_Value()) { failure |= verify_Value_for(n); }
+ if (is_verify_Ideal()) { failure |= verify_Ideal_for(n, false); }
+ if (is_verify_Ideal()) { failure |= verify_Ideal_for(n, true); }
+ if (is_verify_Identity()) { failure |= verify_Identity_for(n); }
// traverse all inputs and outputs
for (uint i = 0; i < n->req(); i++) {
if (n->in(i) != nullptr) {
@@ -1097,6 +1104,27 @@ void PhaseIterGVN::verify_optimize() {
// in the verification code above if that is not possible for some reason (like Load nodes).
assert(!failure, "Missed optimization opportunity in PhaseIterGVN");
}
+
+ verify_empty_worklist(nullptr);
+}
+
+void PhaseIterGVN::verify_empty_worklist(Node* node) {
+ // Verify that the igvn worklist is empty. If no optimization happened, then
+ // nothing needs to be on the worklist.
+ if (_worklist.size() == 0) { return; }
+
+ stringStream ss; // Print as a block without tty lock.
+ for (uint j = 0; j < _worklist.size(); j++) {
+ Node* n = _worklist.at(j);
+ ss.print("igvn.worklist[%d] ", j);
+ n->dump("\n", false, &ss);
+ }
+ if (_worklist.size() != 0 && node != nullptr) {
+ ss.print_cr("Previously optimized:");
+ node->dump("\n", false, &ss);
+ }
+ tty->print_cr("%s", ss.as_string());
+ assert(false, "igvn worklist must still be empty after verify");
}
// Check that type(n) == n->Value(), return true if we have a failure.
@@ -1104,7 +1132,7 @@ void PhaseIterGVN::verify_optimize() {
// (1) Integer "widen" changes, but the range is the same.
// (2) LoadNode performs deep traversals. Load is not notified for changes far away.
// (3) CmpPNode performs deep traversals if it compares oopptr. CmpP is not notified for changes far away.
-bool PhaseIterGVN::verify_node_value(Node* n) {
+bool PhaseIterGVN::verify_Value_for(Node* n) {
// If we assert inside type(n), because the type is still a null, then maybe
// the node never went through gvn.transform, which would be a bug.
const Type* told = type(n);
@@ -1149,15 +1177,873 @@ bool PhaseIterGVN::verify_node_value(Node* n) {
// after loop-opts, so that should take care of many of these cases.
return false;
}
- tty->cr();
- tty->print_cr("Missed Value optimization:");
- n->dump_bfs(1, nullptr, "");
- tty->print_cr("Current type:");
- told->dump_on(tty);
- tty->cr();
- tty->print_cr("Optimized type:");
- tnew->dump_on(tty);
- tty->cr();
+
+ stringStream ss; // Print as a block without tty lock.
+ ss.cr();
+ ss.print_cr("Missed Value optimization:");
+ n->dump_bfs(1, nullptr, "", &ss);
+ ss.print_cr("Current type:");
+ told->dump_on(&ss);
+ ss.cr();
+ ss.print_cr("Optimized type:");
+ tnew->dump_on(&ss);
+ ss.cr();
+ tty->print_cr("%s", ss.as_string());
+ return true;
+}
+
+// Check that all Ideal optimizations that could be done were done.
+// Returns true if it found missed optimization opportunities and
+// false otherwise (no missed optimization, or skipped verification).
+bool PhaseIterGVN::verify_Ideal_for(Node* n, bool can_reshape) {
+ // First, we check a list of exceptions, where we skip verification,
+ // because there are known cases where Ideal can optimize after IGVN.
+ // Some may be expected and cannot be fixed, and others should be fixed.
+ switch (n->Opcode()) {
+ // RangeCheckNode::Ideal looks up the chain for about 999 nodes
+ // (see "Range-Check scan limit"). So, it is possible that something
+ // is optimized in that input subgraph, and the RangeCheck was not
+ // added to the worklist because it would be too expensive to walk
+ // down the graph for 1000 nodes and put all on the worklist.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xbatch --version
+ case Op_RangeCheck:
+ return false;
+
+ // IfNode::Ideal does:
+ // Node* prev_dom = search_identical(dist, igvn);
+ // which means we seach up the CFG, traversing at most up to a distance.
+ // If anything happens rather far away from the If, we may not put the If
+ // back on the worklist.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_If:
+ return false;
+
+ // IfNode::simple_subsuming
+ // Looks for dominating test that subsumes the current test.
+ // Notification could be difficult because of larger distance.
+ //
+ // Found with:
+ // runtime/exceptionMsgs/ArrayIndexOutOfBoundsException/ArrayIndexOutOfBoundsExceptionTest.java#id1
+ // -XX:VerifyIterativeGVN=1110
+ case Op_CountedLoopEnd:
+ return false;
+
+ // LongCountedLoopEndNode::Ideal
+ // Probably same issue as above.
+ //
+ // Found with:
+ // compiler/predicates/assertion/TestAssertionPredicates.java#NoLoopPredicationXbatch
+ // -XX:StressLongCountedLoop=2000000 -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ case Op_LongCountedLoopEnd:
+ return false;
+
+ // RegionNode::Ideal does "Skip around the useless IF diamond".
+ // 245 IfTrue === 244
+ // 258 If === 245 257
+ // 259 IfTrue === 258 [[ 263 ]]
+ // 260 IfFalse === 258 [[ 263 ]]
+ // 263 Region === 263 260 259 [[ 263 268 ]]
+ // to
+ // 245 IfTrue === 244
+ // 263 Region === 263 245 _ [[ 263 268 ]]
+ //
+ // "Useless" means that there is no code in either branch of the If.
+ // I found a case where this was not done yet during IGVN.
+ // Why does the Region not get added to IGVN worklist when the If diamond becomes useless?
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_Region:
+ return false;
+
+ // In AddNode::Ideal, we call "commute", which swaps the inputs so
+ // that smaller idx are first. Tracking it back, it led me to
+ // PhaseIdealLoop::remix_address_expressions which swapped the edges.
+ //
+ // Example:
+ // Before PhaseIdealLoop::remix_address_expressions
+ // 154 AddI === _ 12 144
+ // After PhaseIdealLoop::remix_address_expressions
+ // 154 AddI === _ 144 12
+ // After AddNode::Ideal
+ // 154 AddI === _ 12 144
+ //
+ // I suspect that the node should be added to the IGVN worklist after
+ // PhaseIdealLoop::remix_address_expressions
+ //
+ // This is the only case I looked at, there may be others. Found like this:
+ // java -XX:VerifyIterativeGVN=0100 -Xbatch --version
+ //
+ // The following hit the same logic in PhaseIdealLoop::remix_address_expressions.
+ //
+ // Note: currently all of these fail also for other reasons, for example
+ // because of "commute" doing the reordering with the phi below. Once
+ // that is resolved, we can come back to this issue here.
+ //
+ // case Op_AddD:
+ // case Op_AddI:
+ // case Op_AddL:
+ // case Op_AddF:
+ // case Op_MulI:
+ // case Op_MulL:
+ // case Op_MulF:
+ // case Op_MulD:
+ // if (n->in(1)->_idx > n->in(2)->_idx) {
+ // // Expect "commute" to revert this case.
+ // return false;
+ // }
+ // break; // keep verifying
+
+ // AddFNode::Ideal calls "commute", which can reorder the inputs for this:
+ // Check for tight loop increments: Loop-phi of Add of loop-phi
+ // It wants to take the phi into in(1):
+ // 471 Phi === 435 38 390
+ // 390 AddF === _ 471 391
+ //
+ // Other Associative operators are also affected equally.
+ //
+ // Investigate why this does not happen earlier during IGVN.
+ //
+ // Found with:
+ // test/hotspot/jtreg/compiler/loopopts/superword/ReductionPerf.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_AddD:
+ //case Op_AddI: // Also affected for other reasons, see case further down.
+ //case Op_AddL: // Also affected for other reasons, see case further down.
+ case Op_AddF:
+ case Op_MulI:
+ case Op_MulL:
+ case Op_MulF:
+ case Op_MulD:
+ case Op_MinF:
+ case Op_MinD:
+ case Op_MaxF:
+ case Op_MaxD:
+ // XorINode::Ideal
+ // Found with:
+ // compiler/intrinsics/chacha/TestChaCha20.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_XorI:
+ case Op_XorL:
+ // It seems we may have similar issues with the HF cases.
+ // Found with aarch64:
+ // compiler/vectorization/TestFloat16VectorOperations.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_AddHF:
+ case Op_MulHF:
+ case Op_MaxHF:
+ case Op_MinHF:
+ return false;
+
+ // In MulNode::Ideal the edges can be swapped to help value numbering:
+ //
+ // // We are OK if right is a constant, or right is a load and
+ // // left is a non-constant.
+ // if( !(t2->singleton() ||
+ // (in(2)->is_Load() && !(t1->singleton() || in(1)->is_Load())) ) ) {
+ // if( t1->singleton() || // Left input is a constant?
+ // // Otherwise, sort inputs (commutativity) to help value numbering.
+ // (in(1)->_idx > in(2)->_idx) ) {
+ // swap_edges(1, 2);
+ //
+ // Why was this not done earlier during IGVN?
+ //
+ // Found with:
+ // test/hotspot/jtreg/gc/stress/gcbasher/TestGCBasherWithG1.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_AndI:
+ // Same for AndL.
+ // Found with:
+ // compiler/intrinsics/bigInteger/MontgomeryMultiplyTest.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_AndL:
+ return false;
+
+ // SubLNode::Ideal does transform like:
+ // Convert "c1 - (y+c0)" into "(c1-c0) - y"
+ //
+ // In IGVN before verification:
+ // 8423 ConvI2L === _ 3519 [[ 8424 ]] #long:-2
+ // 8422 ConvI2L === _ 8399 [[ 8424 ]] #long:3..256:www
+ // 8424 AddL === _ 8422 8423 [[ 8383 ]] !orig=[8382]
+ // 8016 ConL === 0 [[ 8383 ]] #long:0
+ // 8383 SubL === _ 8016 8424 [[ 8156 ]] !orig=[8154]
+ //
+ // And then in verification:
+ // 8338 ConL === 0 [[ 8339 8424 ]] #long:-2 <----- Was constant folded.
+ // 8422 ConvI2L === _ 8399 [[ 8424 ]] #long:3..256:www
+ // 8424 AddL === _ 8422 8338 [[ 8383 ]] !orig=[8382]
+ // 8016 ConL === 0 [[ 8383 ]] #long:0
+ // 8383 SubL === _ 8016 8424 [[ 8156 ]] !orig=[8154]
+ //
+ // So the form changed from:
+ // c1 - (y + [8423 ConvI2L])
+ // to
+ // c1 - (y + -2)
+ // but the SubL was not added to the IGVN worklist. Investigate why.
+ // There could be other issues too.
+ //
+ // There seems to be a related AddL IGVN optimization that triggers
+ // the same SubL optimization, so investigate that too.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_SubL:
+ return false;
+
+ // SubINode::Ideal does
+ // Convert "x - (y+c0)" into "(x-y) - c0" AND
+ // Convert "c1 - (y+c0)" into "(c1-c0) - y"
+ //
+ // Investigate why this does not yet happen during IGVN.
+ //
+ // Found with:
+ // test/hotspot/jtreg/compiler/c2/IVTest.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_SubI:
+ return false;
+
+ // AddNode::IdealIL does transform like:
+ // Convert x + (con - y) into "(x - y) + con"
+ //
+ // In IGVN before verification:
+ // 8382 ConvI2L
+ // 8381 ConvI2L === _ 791 [[ 8383 ]] #long:0
+ // 8383 SubL === _ 8381 8382
+ // 8168 ConvI2L
+ // 8156 AddL === _ 8168 8383 [[ 8158 ]]
+ //
+ // And then in verification:
+ // 8424 AddL
+ // 8016 ConL === 0 [[ 8383 ]] #long:0 <--- Was constant folded.
+ // 8383 SubL === _ 8016 8424
+ // 8168 ConvI2L
+ // 8156 AddL === _ 8168 8383 [[ 8158 ]]
+ //
+ // So the form changed from:
+ // x + (ConvI2L(0) - [8382 ConvI2L])
+ // to
+ // x + (0 - [8424 AddL])
+ // but the AddL was not added to the IGVN worklist. Investigate why.
+ // There could be other issues, too. For example with "commute", see above.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_AddL:
+ return false;
+
+ // SubTypeCheckNode::Ideal calls SubTypeCheckNode::verify_helper, which does
+ // Node* cmp = phase->transform(new CmpPNode(subklass, in(SuperKlass)));
+ // record_for_cleanup(cmp, phase);
+ // This verification code in the Ideal code creates new nodes, and checks
+ // if they fold in unexpected ways. This means some nodes are created and
+ // added to the worklist, even if the SubTypeCheck is not optimized. This
+ // goes agains the assumption of the verification here, which assumes that
+ // if the node is not optimized, then no new nodes should be created, and
+ // also no nodes should be added to the worklist.
+ // I see two options:
+ // 1) forbid what verify_helper does, because for each Ideal call it
+ // uses memory and that is suboptimal. But it is not clear how that
+ // verification can be done otherwise.
+ // 2) Special case the verification here. Probably the new nodes that
+ // were just created are dead, i.e. they are not connected down to
+ // root. We could verify that, and remove those nodes from the graph
+ // by setting all their inputs to nullptr. And of course we would
+ // have to remove those nodes from the worklist.
+ // Maybe there are other options too, I did not dig much deeper yet.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xbatch --version
+ case Op_SubTypeCheck:
+ return false;
+
+ // LoopLimitNode::Ideal when stride is constant power-of-2, we can do a lowering
+ // to other nodes: Conv, Add, Sub, Mul, And ...
+ //
+ // 107 ConI === 0 [[ ... ]] #int:2
+ // 84 LoadRange === _ 7 83
+ // 50 ConI === 0 [[ ... ]] #int:0
+ // 549 LoopLimit === _ 50 84 107
+ //
+ // I stepped backward, to see how the node was generated, and I found that it was
+ // created in PhaseIdealLoop::exact_limit and not changed since. It is added to the
+ // IGVN worklist. I quickly checked when it goes into LoopLimitNode::Ideal after
+ // that, and it seems we want to skip lowering it until after loop-opts, but never
+ // add call record_for_post_loop_opts_igvn. This would be an easy fix, but there
+ // could be other issues too.
+ //
+ // Fond with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_LoopLimit:
+ return false;
+
+ // PhiNode::Ideal calls split_flow_path, which tries to do this:
+ // "This optimization tries to find two or more inputs of phi with the same constant
+ // value. It then splits them into a separate Phi, and according Region."
+ //
+ // Example:
+ // 130 DecodeN === _ 129
+ // 50 ConP === 0 [[ 18 91 99 18 ]] #null
+ // 18 Phi === 14 50 130 50 [[ 133 ]] #java/lang/Object * Oop:java/lang/Object *
+ //
+ // turns into:
+ //
+ // 50 ConP === 0 [[ 99 91 18 ]] #null
+ // 130 DecodeN === _ 129 [[ 18 ]]
+ // 18 Phi === 14 130 50 [[ 133 ]] #java/lang/Object * Oop:java/lang/Object *
+ //
+ // We would have to investigate why this optimization does not happen during IGVN.
+ // There could also be other issues - I did not investigate further yet.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_Phi:
+ return false;
+
+ // MemBarNode::Ideal does "Eliminate volatile MemBars for scalar replaced objects".
+ // For examle "The allocated object does not escape".
+ //
+ // It seems the difference to earlier calls to MemBarNode::Ideal, is that there
+ // alloc->as_Allocate()->does_not_escape_thread() returned false, but in verification
+ // it returned true. Why does the MemBarStoreStore not get added to the IGVN
+ // worklist when this change happens?
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_MemBarStoreStore:
+ return false;
+
+ // ConvI2LNode::Ideal converts
+ // 648 AddI === _ 583 645 [[ 661 ]]
+ // 661 ConvI2L === _ 648 [[ 664 ]] #long:0..maxint-1:www
+ // into
+ // 772 ConvI2L === _ 645 [[ 773 ]] #long:-120..maxint-61:www
+ // 771 ConvI2L === _ 583 [[ 773 ]] #long:60..120:www
+ // 773 AddL === _ 771 772 [[ ]]
+ //
+ // We have to investigate why this does not happen during IGVN in this case.
+ // There could also be other issues - I did not investigate further yet.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ case Op_ConvI2L:
+ return false;
+
+ // AddNode::IdealIL can do this transform (and similar other ones):
+ // Convert "a*b+a*c into a*(b+c)
+ // The example had AddI(MulI(a, b), MulI(a, c)). Why did this not happen
+ // during IGVN? There was a mutation for one of the MulI, and only
+ // after that the pattern was as needed for the optimization. The MulI
+ // was added to the IGVN worklist, but not the AddI. This probably
+ // can be fixed by adding the correct pattern in add_users_of_use_to_worklist.
+ //
+ // Found with:
+ // test/hotspot/jtreg/compiler/loopopts/superword/ReductionPerf.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_AddI:
+ return false;
+
+ // ArrayCopyNode::Ideal
+ // calls ArrayCopyNode::prepare_array_copy
+ // calls Compile::conv_I2X_index -> is called with sizetype = intcon(0), I think that
+ // is not expected, and we create a range int:0..-1
+ // calls Compile::constrained_convI2L -> creates ConvI2L(intcon(1), int:0..-1)
+ // note: the type is already empty!
+ // calls PhaseIterGVN::transform
+ // calls PhaseIterGVN::transform_old
+ // calls PhaseIterGVN::subsume_node -> subsume ConvI2L with TOP
+ // calls Unique_Node_List::push -> pushes TOP to worklist
+ //
+ // Once we get back to ArrayCopyNode::prepare_array_copy, we get back TOP, and
+ // return false. This means we eventually return nullptr from ArrayCopyNode::Ideal.
+ //
+ // Question: is it ok to push anything to the worklist during ::Ideal, if we will
+ // return nullptr, indicating nothing happened?
+ // Is it smart to do transform in Compile::constrained_convI2L, and then
+ // check for TOP in calls ArrayCopyNode::prepare_array_copy?
+ // Should we just allow TOP to land on the worklist, as an exception?
+ //
+ // Found with:
+ // compiler/arraycopy/TestArrayCopyAsLoadsStores.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_ArrayCopy:
+ return false;
+
+ // CastLLNode::Ideal
+ // calls ConstraintCastNode::optimize_integer_cast -> pushes CastLL through SubL
+ //
+ // Could be a notification issue, where updates inputs of CastLL do not notify
+ // down through SubL to CastLL.
+ //
+ // Found With:
+ // compiler/c2/TestMergeStoresMemorySegment.java#byte-array
+ // -XX:VerifyIterativeGVN=1110
+ case Op_CastLL:
+ return false;
+
+ // Similar case happens to CastII
+ //
+ // Found With:
+ // compiler/c2/TestScalarReplacementMaxLiveNodes.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_CastII:
+ return false;
+
+ // MaxLNode::Ideal
+ // calls AddNode::Ideal
+ // calls commute -> decides to swap edges
+ //
+ // Another notification issue, because we check inputs of inputs?
+ // MaxL -> Phi -> Loop
+ // MaxL -> Phi -> MaxL
+ //
+ // Found with:
+ // compiler/c2/irTests/TestIfMinMax.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_MaxL:
+ case Op_MinL:
+ return false;
+
+ // OrINode::Ideal
+ // calls AddNode::Ideal
+ // calls commute -> left is Load, right not -> commute.
+ //
+ // Not sure why notification does not work here, seems like
+ // the depth is only 1, so it should work. Needs investigation.
+ //
+ // Found with:
+ // compiler/codegen/TestCharVect2.java#id0
+ // -XX:VerifyIterativeGVN=1110
+ case Op_OrI:
+ case Op_OrL:
+ return false;
+
+ // Bool -> constant folded to 1.
+ // Issue with notification?
+ //
+ // Found with:
+ // compiler/c2/irTests/TestVectorizationMismatchedAccess.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_Bool:
+ return false;
+
+ // LShiftLNode::Ideal
+ // Looks at pattern: "(x + x) << c0", converts it to "x << (c0 + 1)"
+ // Probably a notification issue.
+ //
+ // Found with:
+ // compiler/conversions/TestMoveConvI2LOrCastIIThruAddIs.java
+ // -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ case Op_LShiftL:
+ return false;
+
+ // LShiftINode::Ideal
+ // pattern: ((x + con1) << con2) -> x << con2 + con1 << con2
+ // Could be issue with notification of inputs of inputs
+ //
+ // Side-note: should cases like these not be shared between
+ // LShiftI and LShiftL?
+ //
+ // Found with:
+ // compiler/escapeAnalysis/Test6689060.java
+ // -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110 -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ case Op_LShiftI:
+ return false;
+
+ // AddPNode::Ideal seems to do set_req without removing lock first.
+ // Found with various vector tests tier1-tier3.
+ case Op_AddP:
+ return false;
+
+ // StrIndexOfNode::Ideal
+ // Found in tier1-3.
+ case Op_StrIndexOf:
+ case Op_StrIndexOfChar:
+ return false;
+
+ // StrEqualsNode::Identity
+ //
+ // Found (linux x64 only?) with:
+ // serviceability/sa/ClhsdbThreadContext.java
+ // -XX:+UnlockExperimentalVMOptions -XX:LockingMode=1 -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ case Op_StrEquals:
+ return false;
+
+ // AryEqNode::Ideal
+ // Not investigated. Reshapes itself and adds lots of nodes to the worklist.
+ //
+ // Found with:
+ // vmTestbase/vm/mlvm/meth/stress/compiler/i2c_c2i/Test.java
+ // -XX:+UnlockDiagnosticVMOptions -XX:-TieredCompilation -XX:+StressUnstableIfTraps -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ case Op_AryEq:
+ return false;
+
+ // MergeMemNode::Ideal
+ // Found in tier1-3. Did not investigate further yet.
+ case Op_MergeMem:
+ return false;
+
+ // URShiftINode::Ideal
+ // Found in tier1-3. Did not investigate further yet.
+ case Op_URShiftI:
+ return false;
+
+ // CMoveINode::Ideal
+ // Found in tier1-3. Did not investigate further yet.
+ case Op_CMoveI:
+ return false;
+
+ // CmpPNode::Ideal calls isa_const_java_mirror
+ // and generates new constant nodes, even if no progress is made.
+ // We can probably rewrite this so that only types are generated.
+ // It seems that object types are not hashed, we could investigate
+ // if that is an option as well.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=1110 -Xcomp --version
+ case Op_CmpP:
+ return false;
+
+ // MinINode::Ideal
+ // Did not investigate, but there are some patterns that might
+ // need more notification.
+ case Op_MinI:
+ case Op_MaxI: // preemptively removed it as well.
+ return false;
+ }
+
+ if (n->is_Load()) {
+ // LoadNode::Ideal uses tries to find an earlier memory state, and
+ // checks can_see_stored_value for it.
+ //
+ // Investigate why this was not already done during IGVN.
+ // A similar issue happens with Identity.
+ //
+ // There seem to be other cases where loads go up some steps, like
+ // LoadNode::Ideal going up 10x steps to find dominating load.
+ //
+ // Found with:
+ // test/hotspot/jtreg/compiler/arraycopy/TestCloneAccess.java
+ // -XX:VerifyIterativeGVN=1110
+ return false;
+ }
+
+ if (n->is_Store()) {
+ // StoreNode::Ideal can do this:
+ // // Capture an unaliased, unconditional, simple store into an initializer.
+ // // Or, if it is independent of the allocation, hoist it above the allocation.
+ // That replaces the Store with a MergeMem.
+ //
+ // We have to investigate why this does not happen during IGVN in this case.
+ // There could also be other issues - I did not investigate further yet.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=0100 -Xcomp --version
+ return false;
+ }
+
+ if (n->is_Vector()) {
+ // VectorNode::Ideal swaps edges, but only for ops
+ // that are deemed commutable. But swap_edges
+ // requires the hash to be invariant when the edges
+ // are swapped, which is not implemented for these
+ // vector nodes. This seems not to create any trouble
+ // usually, but we can also get graphs where in the
+ // end the nodes are not all commuted, so there is
+ // definitively an issue here.
+ //
+ // Probably we have two options: kill the hash, or
+ // properly make the hash commutation friendly.
+ //
+ // Found with:
+ // compiler/vectorapi/TestMaskedMacroLogicVector.java
+ // -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110 -XX:+UseParallelGC -XX:+UseNUMA
+ return false;
+ }
+
+ if (n->is_Region()) {
+ // LoopNode::Ideal calls RegionNode::Ideal.
+ // CountedLoopNode::Ideal calls RegionNode::Ideal too.
+ // But I got an issue because RegionNode::optimize_trichotomy
+ // then modifies another node, and pushes nodes to the worklist
+ // Not sure if this is ok, modifying another node like that.
+ // Maybe it is, then we need to look into what to do with
+ // the nodes that are now on the worklist, maybe just clear
+ // them out again. But maybe modifying other nodes like that
+ // is also bad design. In the end, we return nullptr for
+ // the current CountedLoop. But the extra nodes on the worklist
+ // trip the asserts later on.
+ //
+ // Found with:
+ // compiler/eliminateAutobox/TestShortBoxing.java
+ // -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ return false;
+ }
+
+ if (n->is_CallJava()) {
+ // CallStaticJavaNode::Ideal
+ // Led to a crash:
+ // assert((is_CallStaticJava() && cg->is_mh_late_inline()) || (is_CallDynamicJava() && cg->is_virtual_late_inline())) failed: mismatch
+ //
+ // Did not investigate yet, could be a bug.
+ // Or maybe it does not expect to be called during verification.
+ //
+ // Found with:
+ // test/jdk/jdk/incubator/vector/VectorRuns.java
+ // -XX:VerifyIterativeGVN=1110
+
+ // CallDynamicJavaNode::Ideal, and I think also for CallStaticJavaNode::Ideal
+ // and possibly their subclasses.
+ // During late inlining it can call CallJavaNode::register_for_late_inline
+ // That means we do more rounds of late inlining, but might fail.
+ // Then we do IGVN again, and register the node again for late inlining.
+ // This creates an endless cycle. Everytime we try late inlining, we
+ // are also creating more nodes, especially SafePoint and MergeMem.
+ // These nodes are immediately rejected when the inlining fails in the
+ // do_late_inline_check, but they still grow the memory, until we hit
+ // the MemLimit and crash.
+ // The assumption here seems that CallDynamicJavaNode::Ideal does not get
+ // called repeatedly, and eventually we terminate. I fear this is not
+ // a great assumption to make. We should investigate more.
+ //
+ // Found with:
+ // compiler/loopopts/superword/TestDependencyOffsets.java#vanilla-U
+ // -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ return false;
+ }
+
+ // The number of nodes shoud not increase.
+ uint old_unique = C->unique();
+
+ Node* i = n->Ideal(this, can_reshape);
+ // If there was no new Idealization, we are probably happy.
+ if (i == nullptr) {
+ if (old_unique < C->unique()) {
+ stringStream ss; // Print as a block without tty lock.
+ ss.cr();
+ ss.print_cr("Ideal optimization did not make progress but created new unused nodes.");
+ ss.print_cr(" old_unique = %d, unique = %d", old_unique, C->unique());
+ n->dump_bfs(1, nullptr, "", &ss);
+ tty->print_cr("%s", ss.as_string());
+ return true;
+ }
+
+ verify_empty_worklist(n);
+
+ // Everything is good.
+ return false;
+ }
+
+ // We just saw a new Idealization which was not done during IGVN.
+ stringStream ss; // Print as a block without tty lock.
+ ss.cr();
+ ss.print_cr("Missed Ideal optimization (can_reshape=%s):", can_reshape ? "true": "false");
+ if (i == n) {
+ ss.print_cr("The node was reshaped by Ideal.");
+ } else {
+ ss.print_cr("The node was replaced by Ideal.");
+ ss.print_cr("Old node:");
+ n->dump_bfs(1, nullptr, "", &ss);
+ }
+ ss.print_cr("The result after Ideal:");
+ i->dump_bfs(1, nullptr, "", &ss);
+ tty->print_cr("%s", ss.as_string());
+ return true;
+}
+
+// Check that all Identity optimizations that could be done were done.
+// Returns true if it found missed optimization opportunities and
+// false otherwise (no missed optimization, or skipped verification).
+bool PhaseIterGVN::verify_Identity_for(Node* n) {
+ // First, we check a list of exceptions, where we skip verification,
+ // because there are known cases where Ideal can optimize after IGVN.
+ // Some may be expected and cannot be fixed, and others should be fixed.
+ switch (n->Opcode()) {
+ // SafePointNode::Identity can remove SafePoints, but wants to wait until
+ // after loopopts:
+ // // Transforming long counted loops requires a safepoint node. Do not
+ // // eliminate a safepoint until loop opts are over.
+ // if (in(0)->is_Proj() && !phase->C->major_progress()) {
+ //
+ // I think the check for major_progress does delay it until after loopopts
+ // but it does not ensure that the node is on the IGVN worklist after
+ // loopopts. I think we should try to instead check for
+ // phase->C->post_loop_opts_phase() and call record_for_post_loop_opts_igvn.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=1000 -Xcomp --version
+ case Op_SafePoint:
+ return false;
+
+ // MergeMemNode::Identity replaces the MergeMem with its base_memory if it
+ // does not record any other memory splits.
+ //
+ // I did not deeply investigate, but it looks like MergeMemNode::Identity
+ // never got called during IGVN for this node, investigate why.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=1000 -Xcomp --version
+ case Op_MergeMem:
+ return false;
+
+ // ConstraintCastNode::Identity finds casts that are the same, except that
+ // the control is "higher up", i.e. dominates. The call goes via
+ // ConstraintCastNode::dominating_cast to PhaseGVN::is_dominator_helper,
+ // which traverses up to 100 idom steps. If anything gets optimized somewhere
+ // away from the cast, but within 100 idom steps, the cast may not be
+ // put on the IGVN worklist any more.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=1000 -Xcomp --version
+ case Op_CastPP:
+ case Op_CastII:
+ case Op_CastLL:
+ return false;
+
+ // Same issue for CheckCastPP, uses ConstraintCastNode::Identity and
+ // checks dominator, which may be changed, but too far up for notification
+ // to work.
+ //
+ // Found with:
+ // compiler/c2/irTests/TestSkeletonPredicates.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_CheckCastPP:
+ return false;
+
+ // In SubNode::Identity, we do:
+ // Convert "(X+Y) - Y" into X and "(X+Y) - X" into Y
+ // In the example, the AddI had an input replaced, the AddI is
+ // added to the IGVN worklist, but the SubI is one link further
+ // down and is not added. I checked add_users_of_use_to_worklist
+ // where I would expect the SubI would be added, and I cannot
+ // find the pattern, only this one:
+ // If changed AddI/SubI inputs, check CmpU for range check optimization.
+ //
+ // Fix this "notification" issue and check if there are any other
+ // issues.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=1000 -Xcomp --version
+ case Op_SubI:
+ case Op_SubL:
+ return false;
+
+ // PhiNode::Identity checks for patterns like:
+ // r = (x != con) ? x : con;
+ // that can be constant folded to "x".
+ //
+ // Call goes through PhiNode::is_cmove_id and CMoveNode::is_cmove_id.
+ // I suspect there was some earlier change to one of the inputs, but
+ // not all relevant outputs were put on the IGVN worklist.
+ //
+ // Found with:
+ // test/hotspot/jtreg/gc/stress/gcbasher/TestGCBasherWithG1.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_Phi:
+ return false;
+
+ // ConvI2LNode::Identity does
+ // convert I2L(L2I(x)) => x
+ //
+ // Investigate why this did not already happen during IGVN.
+ //
+ // Found with:
+ // compiler/loopopts/superword/TestDependencyOffsets.java#vanilla-A
+ // -XX:VerifyIterativeGVN=1110
+ case Op_ConvI2L:
+ return false;
+
+ // MaxNode::find_identity_operation
+ // Finds patterns like Max(A, Max(A, B)) -> Max(A, B)
+ // This can be a 2-hop search, so maybe notification is not
+ // good enough.
+ //
+ // Found with:
+ // compiler/codegen/TestBooleanVect.java
+ // -XX:VerifyIterativeGVN=1110
+ case Op_MaxL:
+ case Op_MinL:
+ case Op_MaxI:
+ case Op_MinI:
+ case Op_MaxF:
+ case Op_MinF:
+ case Op_MaxHF:
+ case Op_MinHF:
+ case Op_MaxD:
+ case Op_MinD:
+ return false;
+
+
+ // AddINode::Identity
+ // Converts (x-y)+y to x
+ // Could be issue with notification
+ //
+ // Turns out AddL does the same.
+ //
+ // Found with:
+ // compiler/c2/Test6792161.java
+ // -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ case Op_AddI:
+ case Op_AddL:
+ return false;
+
+ // AbsINode::Identity
+ // Not investigated yet.
+ case Op_AbsI:
+ return false;
+ }
+
+ if (n->is_Load()) {
+ // LoadNode::Identity tries to look for an earlier store value via
+ // can_see_stored_value. I found an example where this led to
+ // an Allocation, where we could assume the value was still zero.
+ // So the LoadN can be replaced with a zerocon.
+ //
+ // Investigate why this was not already done during IGVN.
+ // A similar issue happens with Ideal.
+ //
+ // Found with:
+ // java -XX:VerifyIterativeGVN=1000 -Xcomp --version
+ return false;
+ }
+
+ if (n->is_Store()) {
+ // StoreNode::Identity
+ // Not investigated, but found missing optimization for StoreI.
+ // Looks like a StoreI is replaced with an InitializeNode.
+ //
+ // Found with:
+ // applications/ctw/modules/java_base_2.java
+ // -ea -esa -XX:CompileThreshold=100 -XX:+UnlockExperimentalVMOptions -server -XX:-TieredCompilation -Djava.awt.headless=true -XX:+IgnoreUnrecognizedVMOptions -XX:VerifyIterativeGVN=1110
+ return false;
+ }
+
+ if (n->is_Vector()) {
+ // Found with tier1-3. Not investigated yet.
+ // The observed issue was with AndVNode::Identity
+ return false;
+ }
+
+ Node* i = n->Identity(this);
+ // If we cannot find any other Identity, we are happy.
+ if (i == n) {
+ verify_empty_worklist(n);
+ return false;
+ }
+
+ // The verification just found a new Identity that was not found during IGVN.
+ stringStream ss; // Print as a block without tty lock.
+ ss.cr();
+ ss.print_cr("Missed Identity optimization:");
+ ss.print_cr("Old node:");
+ n->dump_bfs(1, nullptr, "", &ss);
+ ss.print_cr("New node:");
+ i->dump_bfs(1, nullptr, "", &ss);
+ tty->print_cr("%s", ss.as_string());
return true;
}
#endif
@@ -1890,12 +2776,12 @@ void PhaseCCP::analyze() {
#ifdef ASSERT
// For every node n on verify list, check if type(n) == n->Value()
-// We have a list of exceptions, see comments in verify_node_value.
+// We have a list of exceptions, see comments in verify_Value_for.
void PhaseCCP::verify_analyze(Unique_Node_List& worklist_verify) {
bool failure = false;
while (worklist_verify.size()) {
Node* n = worklist_verify.pop();
- failure |= verify_node_value(n);
+ failure |= verify_Value_for(n);
}
// If we get this assert, check why the reported nodes were not processed again in CCP.
// We should either make sure that these nodes are properly added back to the CCP worklist
diff --git a/src/hotspot/share/opto/phaseX.hpp b/src/hotspot/share/opto/phaseX.hpp
index c2a0f0dbb77..648e911e783 100644
--- a/src/hotspot/share/opto/phaseX.hpp
+++ b/src/hotspot/share/opto/phaseX.hpp
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1997, 2025, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -494,7 +494,10 @@ class PhaseIterGVN : public PhaseGVN {
void optimize();
#ifdef ASSERT
void verify_optimize();
- bool verify_node_value(Node* n);
+ bool verify_Value_for(Node* n);
+ bool verify_Ideal_for(Node* n, bool can_reshape);
+ bool verify_Identity_for(Node* n);
+ void verify_empty_worklist(Node* n);
#endif
#ifndef PRODUCT
@@ -593,6 +596,14 @@ class PhaseIterGVN : public PhaseGVN {
// '-XX:VerifyIterativeGVN=10'
return ((VerifyIterativeGVN % 100) / 10) == 1;
}
+ static bool is_verify_Ideal() {
+ // '-XX:VerifyIterativeGVN=100'
+ return ((VerifyIterativeGVN % 1000) / 100) == 1;
+ }
+ static bool is_verify_Identity() {
+ // '-XX:VerifyIterativeGVN=1000'
+ return ((VerifyIterativeGVN % 10000) / 1000) == 1;
+ }
protected:
// Sub-quadratic implementation of '-XX:VerifyIterativeGVN=1' (Use-Def verification).
julong _verify_counter;
diff --git a/src/hotspot/share/opto/phasetype.hpp b/src/hotspot/share/opto/phasetype.hpp
index 2546b319762..517f0aa72c7 100644
--- a/src/hotspot/share/opto/phasetype.hpp
+++ b/src/hotspot/share/opto/phasetype.hpp
@@ -47,12 +47,15 @@
flags(EXPAND_VBOX, "Expand VectorBox") \
flags(ELIMINATE_VBOX_ALLOC, "Eliminate VectorBoxAllocate") \
flags(ITER_GVN_BEFORE_EA, "Iter GVN before EA") \
- flags(ITER_GVN_AFTER_VECTOR, "Iter GVN after vector box elimination") \
+ flags(ITER_GVN_AFTER_VECTOR, "Iter GVN after Vector Box Elimination") \
flags(BEFORE_LOOP_OPTS, "Before Loop Optimizations") \
- flags(BEFORE_BEAUTIFY_LOOPS, "Before beautify loops") \
- flags(AFTER_BEAUTIFY_LOOPS, "After beautify loops") \
- flags(BEFORE_LOOP_UNROLLING, "Before Loop Unrolling") \
- flags(AFTER_LOOP_UNROLLING, "After Loop Unrolling") \
+ flags(PHASEIDEAL_BEFORE_EA, "PhaseIdealLoop before EA") \
+ flags(AFTER_EA, "After Escape Analysis") \
+ flags(ITER_GVN_AFTER_EA, "Iter GVN after EA") \
+ flags(BEFORE_BEAUTIFY_LOOPS, "Before Beautify Loops") \
+ flags(AFTER_BEAUTIFY_LOOPS, "After Beautify Loops") \
+ flags(BEFORE_CLOOPS, "Before CountedLoop") \
+ flags(AFTER_CLOOPS, "After CountedLoop") \
flags(BEFORE_SPLIT_IF, "Before Split-If") \
flags(AFTER_SPLIT_IF, "After Split-If") \
flags(BEFORE_LOOP_PREDICATION_IC, "Before Loop Predication IC") \
@@ -69,22 +72,19 @@
flags(AFTER_LOOP_MULTIVERSIONING, "After Loop Multiversioning") \
flags(BEFORE_RANGE_CHECK_ELIMINATION, "Before Range Check Elimination") \
flags(AFTER_RANGE_CHECK_ELIMINATION, "After Range Check Elimination") \
+ flags(ITER_GVN_AFTER_ELIMINATION, "Iter GVN after Eliminating Allocations and Locks") \
flags(BEFORE_PRE_MAIN_POST, "Before Pre/Main/Post Loops") \
flags(AFTER_PRE_MAIN_POST, "After Pre/Main/Post Loops") \
- flags(AUTO_VECTORIZATION1_BEFORE_APPLY, "AutoVectorization 1, Before Apply") \
- flags(AUTO_VECTORIZATION2_AFTER_REORDER, "AutoVectorization 2, After Apply Memop Reordering") \
- flags(AUTO_VECTORIZATION3_AFTER_ADJUST_LIMIT, "AutoVectorization 3, After Adjusting Pre-Loop Limit") \
- flags(AUTO_VECTORIZATION4_AFTER_SPECULATIVE_RUNTIME_CHECKS, "AutoVectorization 4, After Adding Speculative Runtime Checks") \
- flags(AUTO_VECTORIZATION5_AFTER_APPLY, "AutoVectorization 5, After Apply") \
- flags(BEFORE_CLOOPS, "Before CountedLoop") \
- flags(AFTER_CLOOPS, "After CountedLoop") \
- flags(PHASEIDEAL_BEFORE_EA, "PhaseIdealLoop before EA") \
- flags(AFTER_EA, "After Escape Analysis") \
- flags(ITER_GVN_AFTER_EA, "Iter GVN after EA") \
- flags(ITER_GVN_AFTER_ELIMINATION, "Iter GVN after eliminating allocations and locks") \
+ flags(BEFORE_LOOP_UNROLLING, "Before Loop Unrolling") \
+ flags(AFTER_LOOP_UNROLLING, "After Loop Unrolling") \
flags(PHASEIDEALLOOP1, "PhaseIdealLoop 1") \
flags(PHASEIDEALLOOP2, "PhaseIdealLoop 2") \
flags(PHASEIDEALLOOP3, "PhaseIdealLoop 3") \
+ flags(AUTO_VECTORIZATION1_BEFORE_APPLY, "AutoVectorization 1, before Apply") \
+ flags(AUTO_VECTORIZATION2_AFTER_REORDER, "AutoVectorization 2, after Apply Memop Reordering") \
+ flags(AUTO_VECTORIZATION3_AFTER_ADJUST_LIMIT, "AutoVectorization 3, after Adjusting Pre-loop Limit") \
+ flags(AUTO_VECTORIZATION4_AFTER_SPECULATIVE_RUNTIME_CHECKS, "AutoVectorization 4, after Adding Speculative Runtime Checks") \
+ flags(AUTO_VECTORIZATION5_AFTER_APPLY, "AutoVectorization 5, after Apply") \
flags(BEFORE_CCP1, "Before PhaseCCP 1") \
flags(CCP1, "PhaseCCP 1") \
flags(ITER_GVN2, "Iter GVN 2") \
@@ -94,26 +94,26 @@
flags(BEFORE_MACRO_EXPANSION , "Before Macro Expansion") \
flags(AFTER_MACRO_EXPANSION_STEP, "After Macro Expansion Step") \
flags(AFTER_MACRO_EXPANSION, "After Macro Expansion") \
- flags(BARRIER_EXPANSION, "Barrier expand") \
- flags(OPTIMIZE_FINISHED, "Optimize finished") \
- flags(BEFORE_MATCHING, "Before matching") \
- flags(MATCHING, "After matching") \
- flags(GLOBAL_CODE_MOTION, "Global code motion") \
- flags(INITIAL_LIVENESS, "Initial liveness") \
- flags(LIVE_RANGE_STRETCHING, "Live range stretching") \
- flags(AGGRESSIVE_COALESCING, "Aggressive coalescing") \
- flags(INITIAL_SPILLING, "Initial spilling") \
- flags(CONSERVATIVE_COALESCING, "Conservative coalescing") \
- flags(ITERATIVE_SPILLING, "Iterative spilling") \
- flags(AFTER_ITERATIVE_SPILLING, "After iterative spilling") \
- flags(POST_ALLOCATION_COPY_REMOVAL, "Post-allocation copy removal") \
- flags(MERGE_MULTI_DEFS, "Merge multiple definitions") \
- flags(FIX_UP_SPILLS, "Fix up spills") \
+ flags(BARRIER_EXPANSION, "Barrier Expand") \
+ flags(OPTIMIZE_FINISHED, "Optimize Finished") \
+ flags(BEFORE_MATCHING, "Before Matching") \
+ flags(MATCHING, "After Matching") \
+ flags(GLOBAL_CODE_MOTION, "Global Code Motion") \
+ flags(INITIAL_LIVENESS, "Initial Liveness") \
+ flags(LIVE_RANGE_STRETCHING, "Live Range Stretching") \
+ flags(AGGRESSIVE_COALESCING, "Aggressive Coalescing") \
+ flags(INITIAL_SPILLING, "Initial Spilling") \
+ flags(CONSERVATIVE_COALESCING, "Conservative Coalescing") \
+ flags(ITERATIVE_SPILLING, "Iterative Spilling") \
+ flags(AFTER_ITERATIVE_SPILLING, "After Iterative Spilling") \
+ flags(POST_ALLOCATION_COPY_REMOVAL, "Post-allocation Copy Removal") \
+ flags(MERGE_MULTI_DEFS, "Merge Multiple Definitions") \
+ flags(FIX_UP_SPILLS, "Fix up Spills") \
flags(REGISTER_ALLOCATION, "Register Allocation") \
flags(BLOCK_ORDERING, "Block Ordering") \
flags(PEEPHOLE, "Peephole") \
- flags(POSTALLOC_EXPAND, "Post-Allocation Expand") \
- flags(MACH_ANALYSIS, "After mach analysis") \
+ flags(POSTALLOC_EXPAND, "Post-allocation Expand") \
+ flags(MACH_ANALYSIS, "After Mach Analysis") \
flags(FINAL_CODE, "Final Code") \
flags(END, "End") \
flags(FAILURE, "Failure") \
diff --git a/src/hotspot/share/opto/printinlining.cpp b/src/hotspot/share/opto/printinlining.cpp
index accc3dcc637..be51c08fcfb 100644
--- a/src/hotspot/share/opto/printinlining.cpp
+++ b/src/hotspot/share/opto/printinlining.cpp
@@ -22,10 +22,10 @@
*
*/
-#include "opto/printinlining.hpp"
-#include "opto/callnode.hpp"
#include "memory/allocation.hpp"
#include "memory/resourceArea.hpp"
+#include "opto/callnode.hpp"
+#include "opto/printinlining.hpp"
bool InlinePrinter::is_enabled() const {
return C->print_intrinsics() || C->print_inlining();
@@ -105,5 +105,6 @@ void InlinePrinter::IPInlineSite::dump(outputStream* tty, int level) const {
_children.visit_in_order([=](auto* node) {
node->val().dump(tty, level + 1);
+ return true;
});
}
diff --git a/src/hotspot/share/opto/printinlining.hpp b/src/hotspot/share/opto/printinlining.hpp
index ae79648319b..57f4b51858e 100644
--- a/src/hotspot/share/opto/printinlining.hpp
+++ b/src/hotspot/share/opto/printinlining.hpp
@@ -26,9 +26,9 @@
#define PRINTINLINING_HPP
#include "memory/allocation.hpp"
-#include "utilities/ostream.hpp"
-#include "utilities/growableArray.hpp"
#include "nmt/nmtTreap.hpp"
+#include "utilities/growableArray.hpp"
+#include "utilities/ostream.hpp"
class JVMState;
class ciMethod;
diff --git a/src/hotspot/share/opto/rangeinference.cpp b/src/hotspot/share/opto/rangeinference.cpp
new file mode 100644
index 00000000000..40b9da4bde5
--- /dev/null
+++ b/src/hotspot/share/opto/rangeinference.cpp
@@ -0,0 +1,1081 @@
+/*
+ * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 only, as
+ * published by the Free Software Foundation.
+ *
+ * This code is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+ * version 2 for more details (a copy is included in the LICENSE file that
+ * accompanied this code).
+ *
+ * You should have received a copy of the GNU General Public License version
+ * 2 along with this work; if not, write to the Free Software Foundation,
+ * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
+ *
+ * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
+ * or visit www.oracle.com if you need additional information or have any
+ * questions.
+ *
+ */
+
+#include "opto/rangeinference.hpp"
+#include "opto/type.hpp"
+#include "utilities/intn_t.hpp"
+#include "utilities/tuple.hpp"
+
+// If the cardinality of a TypeInt is below this threshold, use min widen, see
+// TypeIntPrototype::normalize_widen
+constexpr juint SMALL_TYPEINT_THRESHOLD = 3;
+
+// This represents the result of an iterative calculation
+template
+class AdjustResult {
+public:
+ bool _progress; // whether there is progress compared to the last iteration
+ bool _present; // whether the result is empty, typically due to the calculation arriving at contradiction
+ T _result;
+
+ bool empty() const {
+ return !_present;
+ }
+
+ static AdjustResult make_empty() {
+ return {true, false, {}};
+ }
+};
+
+// This is the result of canonicalizing a simple interval (see TypeInt at
+// type.hpp)
+template
+class SimpleCanonicalResult {
+ static_assert(U(-1) > U(0), "bit info should be unsigned");
+public:
+ const bool _present; // whether this is an empty set
+ const RangeInt _bounds; // The bounds must be in the same half of the integer domain (see TypeInt)
+ const KnownBits _bits;
+
+ SimpleCanonicalResult(bool present, const RangeInt& bounds, const KnownBits& bits)
+ : _present(present), _bounds(bounds), _bits(bits) {
+ if (!present) {
+ return;
+ }
+ // Do some verification
+ assert(bits.is_satisfied_by(bounds._lo) && bits.is_satisfied_by(bounds._hi), "must be canonical");
+ // 0b1000...
+ constexpr U mid_point = (std::numeric_limits::max() >> 1) + U(1);
+ assert((bounds._lo < mid_point) == (bounds._hi < mid_point), "must be a simple interval, see Lemma 4");
+ }
+
+ bool empty() const {
+ return !_present;
+ }
+
+ static SimpleCanonicalResult make_empty() {
+ return SimpleCanonicalResult(false, {}, {});
+ }
+};
+
+// Find the minimum value that is not less than lo and satisfies bits. If there
+// does not exist one such number, the calculation will return a value < lo.
+//
+// Formally, this function tries to find the minimum value that is not less
+// than lo and satisfies bits, assuming such value exists. The cases where such
+// value does not exists automatically follows.
+//
+// If there exists a value not less than lo and satisfies bits, then this
+// function will always find one such value. The converse is also true, that is
+// if this function finds a value not less than lo and satisfies bits, then it
+// must trivially be the case that there exists one such value. As a result,
+// the negation of those statements are also equivalent, there does not exists
+// a value not less than lo and satisfies bits if and only if this function
+// does not return one such value.
+//
+// In practice, since the algorithm always ensures that the returned value
+// satisfies bits, we only need to check if it is not less than lo.
+//
+// Here, we view a number in binary as a bit string. As a result, the first
+// bit refers to the highest bit (the MSB), the last bit refers to the lowest
+// bit (the LSB), a bit comes before (being higher than) another if it is more
+// significant, and a bit comes after (being lower than) another if it is less
+// significant. For a value n with w bits, we denote n[0] the first (highest)
+// bit of n, n[1] the second bit, ..., n[w - 1] the last (lowest) bit of n.
+template
+static U adjust_lo(U lo, const KnownBits& bits) {
+ // Violation of lo with respects to bits
+ // E.g: lo = 1100
+ // zeros = 0100
+ // ones = 1001
+ // zero_violation = 0100, i.e the second bit should be zero, but it is 1 in
+ // lo. Similarly, one_violation = 0001, i.e the last bit should be one, but
+ // it is 0 in lo. These make lo not satisfy the bit constraints, which
+ // results in us having to find the smallest value that satisfies bits.
+ U zero_violation = lo & bits._zeros;
+ U one_violation = ~lo & bits._ones;
+ if (zero_violation == one_violation) {
+ // This means lo does not violate bits, it is the result
+ assert(zero_violation == U(0), "");
+ return lo;
+ }
+
+ /*
+ 1. Intuition:
+ Call r the lowest value not smaller than lo that satisfies bits, consider the
+ first bit in r that is different from the corresponding bit in lo:
+ - Since r is larger than lo the bit must be 0 in lo and 1 in r
+ - Since r must satisify bits the bit must be 0 in zeros
+ - Since r should be the smallest value, this bit should be the lowest one
+ possible
+
+ E.g: 1 2 3 4 5 6
+ lo = 1 0 0 1 1 0
+ x = 1 0 1 0 1 0
+ y = 0 1 1 1 1 1
+ x would be larger than lo since the first different bit is the 3rd one,
+ while y is smaller than lo because the first different bit is the 1st bit.
+ Next, consider:
+ x1 = 1 0 1 0 1 0
+ x2 = 1 0 0 1 1 1
+ Both x1 and x2 are larger than lo, but x1 > x2 since its first different
+ bit from lo is the 3rd one, while with x2 it is the 7th one. As a result,
+ if both x1 and x2 satisfy bits, x2 would be closer to our true result.
+
+ 2. Formality:
+
+ Call r the smallest value not smaller than lo that satisfies bits. Since lo
+ does not satisfy bits, lo < r (2.1)
+
+ Call i the largest bit index such that:
+
+ - lo[x] satisfies bits for 0 <= x < i (2.2)
+ - zeros[i] = 0 (2.3)
+ - lo[i] = 0 (2.4)
+
+ Consider v:
+
+ - v[x] = lo[x], for 0 <= x < i (2.5)
+ - v[i] = 1 (2.6)
+ - v[x] = ones[x], for x > i (2.7)
+
+ We will prove that v == r.
+
+ a. Firstly, we prove that r <= v:
+
+ a.1. lo < v, since:
+ lo[x] == v[x], for 0 <= x < i (according to 2.5)
+ lo[i] < v[i] (according to 2.4 and 2.6, lo[i] == 0 < v[i] == 1)
+ bits at x > i have lower significance, and are thus irrelevant
+
+ a.2. v satisfies bits, because:
+ v[x] satisfies bits for 0 <= x < i (according to 2.2 and 2.5)
+ v[i] satisfies bits:
+ According to 2.3 and 2.6, zeros[i] == 0 and v[i] == 1, v[i] does not violate
+ bits, which means v[i] satisfies bits
+ v[x] satisfies bits for x > i:
+ Assume bits is not contradictory, we cannot have:
+ ones[x] == 1, v[x] == 0 (according to 2.7, v[x] == ones[x])
+ zeros[x] == 1, v[x] == 1 (according to 2.7, ones[x] == v[x] == 1, which means
+ bits is contradictory)
+
+ From a.1 and a.2, v > lo and v satisfies bits. Which means r <= v since r is the
+ smallest such value.
+
+ b. Secondly, from r <= v, we prove that r == v. Suppose the contradiction r < v:
+
+ Since r < v, there must be a bit position j that:
+
+ r[j] == 0 (2.b.1)
+ v[j] == 1 (2.b.2)
+ r[x] == v[x], for x < j (2.b.3)
+
+ b.1. If j < i
+ This means that:
+ r[j] == 0 (according to 2.b.1)
+ lo[j] == 1 (according to 2.b.2 and 2.5, lo[j] == v[j] == 1 because j < i)
+ r[x] == lo[x], for x < j (according to 2.b.3 and 2.5, lo[x] == v[x] == r[x] with x < j < i)
+ bits at x > j have lower significance, and are thus irrelevant
+
+ Which leads to r < lo, which contradicts that lo < r (acording to 2.1)
+
+ b.2. If j == i
+ Since r > lo (according to 2.1), there must exist a bit index k such that:
+
+ r[k] == 1 (2.b.2.1)
+ lo[k] == 0 (2.b.2.2)
+ r[x] == lo[x], for x < k (2.b.2.3)
+
+ Then, since we have:
+ r[x] == v[x], for x < i (according to 2.b.3)
+ v[x] == lo[x], for x < i (according to 2.5)
+ r[i] == 0 (according to 2.b.1 because i == j)
+ lo[i] == 0 (according to 2.4)
+
+ this leads to: r[x] == lo[x], for x <= i
+ while r[k] == 1 != lo[k] == 0, we can conclude that k > i
+
+ However, since:
+ lo[x] satisfies bits for 0 <= x < k:
+ According to 2.b.2.3, lo[x] == r[x] and r satisfies bits
+ zeros[k] == 0 (according to 2.b.2.1, r[k] == 1 and r satisfies bits)
+ lo[k] == 0 (according to 2.b.2.2)
+
+ This contradicts the assumption that i is the largest bit index satisfying such conditions.
+
+ b.3. If j > i
+ ones[j] == v[j] (according to 2.7 since j > i)
+ v[j] == 1 (according to 2.b.2)
+ r[j] == 0 (according to 2.b.1)
+
+ This means that r[j] == 0 and ones[j] == 1, this contradicts the assumption that r
+ satisfies bits.
+
+ All cases lead to contradictions, which mean r < v is incorrect, which means that
+ r == v, which means the value v having the above form is the lowest value not smaller
+ than lo that satisfies bits.
+
+ 3. Conclusion
+ Our objective now is to find the largest value i that satisfies:
+ - lo[x] satisfies bits for 0 <= x < i (3.1)
+ - zeros[i] = 0 (3.2)
+ - lo[i] = 0 (3.3)
+ */
+
+ // The algorithm depends on whether the first violation violates zeros or
+ // ones. If it violates zeros, we have the bit being 1 in zero_violation and
+ // 0 in one_violation. Since all higher bits are 0 in zero_violation and
+ // one_violation, we have zero_violation > one_violation. Similarly, if the
+ // first violation violates ones, we have zero_violation < one_violation.
+ if (zero_violation < one_violation) {
+ // This means that the first bit that does not satisfy the bit requirement
+ // is a 0 that should be a 1. Obviously, since the bit at that position in
+ // ones is 1, the same bit in zeros is 0.
+ //
+ // From section 3 above, we know i is the largest bit index such that:
+ // - lo[x] satisfies bits for 0 <= x < i (3.1)
+ // - zeros[i] = 0 (3.2)
+ // - lo[i] = 0 (3.3)
+ //
+ // For the given i, we know that lo satisfies all bits before i, hence (3.1)
+ // holds. Further, lo[i] = 0 (3.3), and we have a one violation at i, hence
+ // zero[i] = 0 (3.2). Any smaller i would not be the largest possible such
+ // index. Any larger i would violate (3.1), since lo[i] does not satisfy bits.
+ // As a result, the first violation is the bit i we are looking for.
+ //
+ // E.g: 1 2 3 4 5 6 7 8
+ // lo = 1 1 0 0 0 1 1 0
+ // zeros = 0 0 1 0 0 1 0 0
+ // ones = 0 1 0 1 0 0 1 0
+ // 1-vio = 0 0 0 1 0 0 0 0
+ // 0-vio = 0 0 0 0 0 1 0 0
+ // Since the result must have the 4th bit set, it must be at least:
+ // 1 1 0 1 0 0 0 0
+ // This value must satisfy zeros, because all bits before the 4th bit have
+ // already satisfied zeros, and all bits after the 4th bit are all 0 now.
+ // Just OR this value with ones to obtain the final result.
+
+ // first_violation is the position of the violation counting from the
+ // highest bit down (0-based), since i == 4, first_violation == 3
+ juint first_violation = count_leading_zeros(one_violation);
+ // 1 0 0 0 0 0 0 0
+ constexpr U highest_bit = (std::numeric_limits::max() >> 1) + U(1);
+ // This is the bit at which we want to change the bit 0 in lo to a 1, and
+ // all bits after to zero. This is similar to an operation that aligns lo
+ // up to the next multiple of this modulo.
+ // 0 0 0 1 0 0 0 0
+ U alignment = highest_bit >> first_violation;
+ // This is the first value which have the violated bit being 1, which means
+ // that the result should not be smaller than this.
+ // This is a standard operation to align a value up to the next multiple of
+ // a certain power of 2. Since alignment is a power of 2, -alignment is a
+ // value having all the bits being 1 upto the location of the bit in
+ // alignment (in the example, -alignment = 11110000). As a result,
+ // lo & -alignment set all bits after the bit in alignment to 0, which is
+ // equivalent to rounding lo down to a multiple of alignment. To round lo
+ // up to the next multiple of alignment, we add alignment to the rounded
+ // down value.
+ // Note that this computation cannot overflow as the bit in lo that is at
+ // the same position as the only bit 1 in alignment must be 0. As a result,
+ // this operation just set that bit to 1 and set all the bits after to 0.
+ // We now have:
+ // - new_lo[x] = lo[x], for 0 <= x < i (2.5)
+ // - new_lo[i] = 1 (2.6)
+ // - new_lo[x] = 0, for x > i (not yet 2.7)
+ // 1 1 0 1 0 0 0 0
+ U new_lo = (lo & -alignment) + alignment;
+ // Note that there exists no value x not larger than i such that
+ // new_lo[x] == 0 and ones[x] == 1. This is because all bits of lo before i
+ // should satisfy bits, and new_lo[i] == 1. As a result, doing
+ // new_lo |= bits.ones will give us a value such that:
+ // - new_lo[x] = lo[x], for 0 <= x < i (2.5)
+ // - new_lo[i] = 1 (2.6)
+ // - new_lo[x] = ones[x], for x > i (2.7)
+ // This is the result we are looking for.
+ // 1 1 0 1 0 0 1 0
+ new_lo |= bits._ones;
+ // Note that in this case, new_lo is always a valid answer. That is, it is
+ // a value not less than lo and satisfies bits.
+ assert(lo < new_lo, "the result must be valid");
+ return new_lo;
+ } else {
+ assert(zero_violation > one_violation, "remaining case");
+ // This means that the first bit that does not satisfy the bit requirement
+ // is a 1 that should be a 0.
+ //
+ // From section 3 above, we know i is the largest bit index such that:
+ // - lo[x] satisfies bits for 0 <= x < i (3.1)
+ // - zeros[i] = 0 (3.2)
+ // - lo[i] = 0 (3.3)
+ //
+ // We know that lo satisfies all bits before first_violation, hence (3.1)
+ // holds. However, first_violation is not the value i we are looking for
+ // because lo[first_violation] == 1. We can also see that any larger value
+ // of i would violate (3.1) since lo[first_violation] does not satisfy
+ // bits. As a result, we should find the last index x upto first_violation
+ // such that lo[x] == zeros[x] == 0. That value of x would be the value of
+ // i we are looking for.
+ //
+ // E.g: 1 2 3 4 5 6 7 8
+ // lo = 1 0 0 0 1 1 1 0
+ // zeros = 0 0 0 1 0 1 0 0
+ // ones = 1 0 0 0 0 0 1 1
+ // 1-vio = 0 0 0 0 0 0 0 1
+ // 0-vio = 0 0 0 0 0 1 0 0
+ // The first violation is the 6th bit, which should be 0. We want to flip
+ // it to 0. However, since we must obtain a value larger than lo, we must
+ // find an earlier bit that can be flipped from 0 to 1. The 5th cannot be
+ // the bit we are looking for, because it is already 1, the 4th bit also
+ // cannot be, because it must be 0. As a result, the last bit we can flip,
+ // which is the first different bit between the result and lo must be the
+ // 3rd bit. As a result, the result must not be smaller than:
+ // 1 0 1 0 0 0 0 0
+ // This one satisfies zeros so we can use the logic in the previous case,
+ // just OR with ones to obtain the final result, which is:
+ // 1 0 1 0 0 0 1 1
+
+ juint first_violation = count_leading_zeros(zero_violation);
+ // This masks out all bits after the first violation
+ // 1 1 1 1 1 0 0 0
+ U find_mask = ~(std::numeric_limits::max() >> first_violation);
+ // We want to find the last index x upto first_violation such that
+ // lo[x] == zeros[x] == 0.
+ // We start with all bits where lo[x] == zeros[x] == 0:
+ // 0 1 1 0 0 0 0 1
+ U neither = ~(lo | bits._zeros);
+ // Now let us find all the bit indices x upto first_violation such that
+ // lo[x] == zeros[x] == 0. The last one of these bits must be at index i.
+ // 0 1 1 0 0 0 0 0
+ U neither_upto_first_violation = neither & find_mask;
+ // We now want to select the last one of these candidates, which is exactly
+ // the last index x upto first_violation such that lo[x] == zeros[x] == 0.
+ // This would be the value i we are looking for.
+ // Similar to the other case, we want to obtain the value with only the bit
+ // i set, this is equivalent to extracting the last set bit of
+ // neither_upto_first_violation, do it directly without going through i.
+ // The formula x & (-x) will give us the last set bit of an integer x
+ // (please see the x86 instruction blsi).
+ // In our example, i == 2
+ // 0 0 1 0 0 0 0 0
+ U alignment = neither_upto_first_violation & (-neither_upto_first_violation);
+ // Set the bit of lo at i and unset all the bits after, this is the
+ // smallest value that satisfies bits._zeros. Similar to the above case,
+ // this is similar to aligning lo up to the next multiple of alignment.
+ // Also similar to the above case, this computation cannot overflow.
+ // We now have:
+ // - new_lo[x] = lo[x], for 0 <= x < i (2.5)
+ // - new_lo[i] = 1 (2.6)
+ // - new_lo[x] = 0, for x > i (not yet 2.7)
+ // 1 0 1 0 0 0 0 0
+ U new_lo = (lo & -alignment) + alignment;
+ // Note that there exists no value x not larger than i such that
+ // new_lo[x] == 0 and ones[x] == 1. This is because all bits of lo before i
+ // should satisfy bits, and new_lo[i] == 1. As a result, doing
+ // new_lo |= bits.ones will give us a value such that:
+ // - new_lo[x] = lo[x], for 0 <= x < i (2.5)
+ // - new_lo[i] = 1 (2.6)
+ // - new_lo[x] = ones[x], for x > i (2.7)
+ // This is the result we are looking for.
+ // 1 0 1 0 0 0 1 1
+ new_lo |= bits._ones;
+ // Note that formally, this function assumes that there exists a value not
+ // smaller than lo and satisfies bits. This implies the existence of the
+ // index i satisfies (3.1-3.3), which means that
+ // neither_upto_first_violation != 0. The converse is
+ // also true, if neither_upto_first_violation != 0, then an index i
+ // satisfies (3.1-3.3) exists, which implies the existence of a value not
+ // smaller than lo and satisfies bits. As a result, the negation of those
+ // statements are equivalent. neither_upto_first_violation == 0 if and only
+ // if there does not exists a value not smaller than lo and satisfies bits.
+ // In this case, alignment == 0 and new_lo == bits._ones. We know that, if
+ // the assumption of this function holds, we return a value satisfying
+ // bits, and if the assumption of this function does not hold, the returned
+ // value would be bits._ones, which also satisfies bits. As a result, this
+ // function always returns a value satisfying bits, regardless whether if
+ // the assumption of this function holds. In conclusion, the caller only
+ // needs to check lo <= new_lo to find the cases where there exists no
+ // value not smaller than lo and satisfies bits (see the overview of the
+ // function).
+ assert(lo < new_lo || new_lo == bits._ones, "invalid result must be bits._ones");
+ return new_lo;
+ }
+}
+
+// Try to tighten the bound constraints from the known bit information. I.e, we
+// find the smallest value not smaller than lo, as well as the largest value
+// not larger than hi both of which satisfy bits
+// E.g: lo = 0010, hi = 1001
+// zeros = 0011
+// ones = 0000
+// -> 4-aligned
+//
+// 0 1 2 3 4 5 6 7 8 9 10
+// 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010
+// bits: ok . . . ok . . . ok . .
+// bounds: lo hi
+// adjust: --------> lo hi <---
+template
+static AdjustResult>
+adjust_unsigned_bounds_from_bits(const RangeInt& bounds, const KnownBits& bits) {
+ U new_lo = adjust_lo(bounds._lo, bits);
+ if (new_lo < bounds._lo) {
+ // This means we wrapped around, which means no value not less than lo
+ // satisfies bits
+ return AdjustResult>::make_empty();
+ }
+
+ // We need to find the largest value not larger than hi that satisfies bits
+ // One possible method is to do similar to adjust_lo, just with the other
+ // direction
+ // However, we can observe that if v satisfies {bits._zeros, bits._ones},
+ // then ~v would satisfy {bits._ones, bits._zeros}. Combine with the fact
+ // that bitwise-not is a strictly decreasing function, if new_hi is the
+ // largest value not larger than hi that satisfies {bits._zeros, bits._ones},
+ // then ~new_hi is the smallest value not smaller than ~hi that satisfies
+ // {bits._ones, bits._zeros}.
+ //
+ // Proof:
+ // Calling h the smallest value not smaller than ~hi that satisfies
+ // {bits._ones, bits._zeros}.
+ //
+ // 1. Since h satisfies {bits._ones, bits._zeros}, ~h satisfies
+ // {bits._zeros, bits._ones}. Assume the contradiction ~h does not satisfy
+ // {bits._zeros, bits._ones}, There can be 2 cases:
+ // 1.1. There is a bit in ~h that is 0 where the corresponding bit in ones
+ // is 1. This implies the corresponding bit in h is 1. But this is
+ // contradictory since h satisfies {bits._ones, bits._zeros}.
+ // 1.2. There is a bit in ~h that is 1 where the corresponding bit in zeros
+ // is 1. Similarly, this leads to contradiction because h needs to
+ // satisfy {bits._ones, bits._zeros}.
+ //
+ // 2. Assume there is a value k that is larger than ~h such that k is not
+ // larger than hi, i.e. ~h < k <= hi, and k satisfies {bits._zeros, bits._ones}.
+ // As a result, ~k would satisfy {bits._ones, bits._zeros}. And since bitwise-not
+ // is a strictly decreasing function, given ~h < k <= hi, we have h > ~k >= ~hi.
+ // This contradicts the assumption that h is the smallest value not smaller than
+ // ~hi and satisfies {bits._ones, bits._zeros}.
+ //
+ // As a result, ~h is the largest value not larger than hi that satisfies
+ // bits (QED).
+ U h = adjust_lo(~bounds._hi, {bits._ones, bits._zeros});
+ if (h < ~bounds._hi) {
+ return AdjustResult>::make_empty();
+ }
+
+ U new_hi = ~h;
+ bool progress = (new_lo != bounds._lo) || (new_hi != bounds._hi);
+ bool present = new_lo <= new_hi;
+ return {progress, present, {new_lo, new_hi}};
+}
+
+// Try to tighten the known bit constraints from the bound information by
+// extracting the common prefix of lo and hi and combining with the current
+// bit constraints
+// E.g: lo = 010011
+// hi = 010100,
+// then all values in [lo, hi] would be
+// 010***
+template
+static AdjustResult>
+adjust_bits_from_unsigned_bounds(const KnownBits& bits, const RangeInt& bounds) {
+ // Find the mask to filter the common prefix, all values between bounds._lo
+ // and bounds._hi should share this common prefix in terms of bits
+ U mismatch = bounds._lo ^ bounds._hi;
+ // Find the first mismatch, all bits before it are the same in bounds._lo and
+ // bounds._hi
+ U match_mask = mismatch == U(0) ? std::numeric_limits