macOS
a continuously updated machine running macOS. See the section on Apple Xcode on some strategies to deal with this. -It is recommended that you use at least macOS 13 (Ventura) and Xcode -14, but earlier versions may also work.
+It is recommended that you use at least macOS 14 and Xcode 15.4, but +earlier versions may also work.
The standard macOS environment contains the basic tooling needed to build, but for external libraries a package manager is recommended. The JDK uses homebrew in the examples, but @@ -590,11 +616,11 @@
Native Compiler
HotSpot Coding Style
- Introduction
- Structure and Formatting
@@ -99,6 +100,21 @@
Introduction
existing HotSpot code, making it easier to read and maintain. Failure to follow these guidelines may lead to discussion during code reviews, if not outright rejection of a change. +Changing this Document
+Proposed changes should be discussed on the HotSpot Developers mailing +list. Changes are likely to be cautious and incremental, since HotSpot +coders have been using these guidelines for years.
+Substantive changes are approved by rough consensus +of the HotSpot Group +Members. The Group Lead determines whether consensus has been +reached.
+Editorial changes (changes that only affect the description of +HotSpot style, not its substance) do not require the full consensus +gathering process. The normal HotSpot pull request process may be used +for editorial changes, with the additional requirement that the +requisite reviewers are also HotSpot Group Members.
Why Care About Style?
Some programmers seem to have lexers and even C preprocessors installed directly behind their eyeballs. The rest of us require code @@ -124,7 +140,7 @@
Why Care About Style?
reformatting the whole thing. Also consider separating changes that make extensive stylistic updates from those which make functional changes. -Counterexamples and Updates
+Counterexamples
Many of the guidelines mentioned here have (sometimes widespread) counterexamples in the HotSpot code base. Finding a counterexample is not sufficient justification for new code to follow the counterexample @@ -137,20 +153,6 @@
Counterexamples and Updates
of course, is "When in Rome do as the Romans". Sometimes in the suburbs of Rome the rules are a little different; these differences can be pointed out here. -Proposed changes should be discussed on the HotSpot Developers mailing -list. Changes are likely to be cautious and incremental, since HotSpot -coders have been using these guidelines for years.
-Substantive changes are approved by rough consensus -of the HotSpot Group -Members. The Group Lead determines whether consensus has been -reached.
-Editorial changes (changes that only affect the description of -HotSpot style, not its substance) do not require the full consensus -gathering process. The normal HotSpot pull request process may be used -for editorial changes, with the additional requirement that the -requisite reviewers are also HotSpot Group Members.
Structure and Formatting
Factoring and Class Design
-
diff --git a/doc/hotspot-style.md b/doc/hotspot-style.md
index 0a135b2074c..f4348bea51d 100644
--- a/doc/hotspot-style.md
+++ b/doc/hotspot-style.md
@@ -8,6 +8,24 @@ HotSpot code, making it easier to read and maintain. Failure to
follow these guidelines may lead to discussion during code reviews, if
not outright rejection of a change.
+### Changing this Document
+
+Proposed changes should be discussed on the
+[HotSpot Developers](mailto:hotspot-dev@openjdk.org) mailing
+list. Changes are likely to be cautious and incremental, since HotSpot
+coders have been using these guidelines for years.
+
+Substantive changes are approved by
+[rough consensus](https://www.rfc-editor.org/rfc/rfc7282.html) of
+the [HotSpot Group](https://openjdk.org/census#hotspot) Members.
+The Group Lead determines whether consensus has been reached.
+
+Editorial changes (changes that only affect the description of HotSpot
+style, not its substance) do not require the full consensus gathering
+process. The normal HotSpot pull request process may be used for
+editorial changes, with the additional requirement that the requisite
+reviewers are also HotSpot Group Members.
+
### Why Care About Style?
Some programmers seem to have lexers and even C preprocessors
@@ -38,7 +56,7 @@ reformatting the whole thing. Also consider separating changes that
make extensive stylistic updates from those which make functional
changes.
-### Counterexamples and Updates
+### Counterexamples
Many of the guidelines mentioned here have (sometimes widespread)
counterexamples in the HotSpot code base. Finding a counterexample is
@@ -54,22 +72,6 @@ rule, of course, is "When in Rome do as the Romans". Sometimes in the
suburbs of Rome the rules are a little different; these differences
can be pointed out here.
-Proposed changes should be discussed on the
-[HotSpot Developers](mailto:hotspot-dev@openjdk.org) mailing
-list. Changes are likely to be cautious and incremental, since HotSpot
-coders have been using these guidelines for years.
-
-Substantive changes are approved by
-[rough consensus](https://www.rfc-editor.org/rfc/rfc7282.html) of
-the [HotSpot Group](https://openjdk.org/census#hotspot) Members.
-The Group Lead determines whether consensus has been reached.
-
-Editorial changes (changes that only affect the description of HotSpot
-style, not its substance) do not require the full consensus gathering
-process. The normal HotSpot pull request process may be used for
-editorial changes, with the additional requirement that the requisite
-reviewers are also HotSpot Group Members.
-
## Structure and Formatting
### Factoring and Class Design
diff --git a/doc/hotspot-unit-tests.html b/doc/hotspot-unit-tests.html
index fcd4a93f8e4..b3700abb3ac 100644
--- a/doc/hotspot-unit-tests.html
+++ b/doc/hotspot-unit-tests.html
@@ -189,7 +189,7 @@
Changing a flag’s value could break the invariants between flags' +
Changing a flag's value could break the invariants between flags' values and hence could lead to unexpected/unsupported JVM state.
+ + + +FLAG_SET_*macros can change more than one flag (in diff --git a/doc/starting-next-release.html b/doc/starting-next-release.html new file mode 100644 index 00000000000..421229f9fbc --- /dev/null +++ b/doc/starting-next-release.html @@ -0,0 +1,127 @@ + + +Explanation of start of release changes + + + + + ++ + +Explanation of start of release changes
+Overview
+The start of release changes, the changes that turn JDK N +into JDK (N+1), are primarily small updates to various files +along with new files to store symbol information to allow +
+javac --release N ...to run on JDK (N+1).The updates include changes to files holding meta-data about the +release, files under the
+srcdirectory for API and tooling +updates, and incidental updates under thetest+directory.Details and file updates
+As a matter of policy, there are a number of semantically distinct +concepts which get incremented separately at the start of a new +release:
+-
+
- Feature value of
Runtime.version()
+ - Highest source version modeled by
+
javax.lang.model.SourceVersion
+ - Highest class file format major version recognized by the +platform +
- Highest
+
-source/-target/--release+argument recognized byjavacand related tools
+
The expected file updates are listed below. Additional files may need +to be updated for a particular release.
+Meta-data files
+-
+
jcheck/conf: update meta-data used by +jcheckand the Skara tooling
+make/conf/version-numbers.conf: update to meta-data +used in the build
+
+srcfiles-
+
src/hotspot/share/classfile/classFileParser.cpp: add a +#definefor the new version
+src/java.base/share/classes/java/lang/classfile/ClassFile.java: +add a constant for the new class file format version
+src/java.base/share/classes/java/lang/reflect/ClassFileFormatVersion.java: +add anenumconstant for the new class file format +version
+src/java.compiler/share/classes/javax/lang/model/SourceVersion.java: +add anenumconstant for the new source version
+src/java.compiler/share/classes/javax/lang/model/util/*+visitors: Update@SupportedSourceVersionannotations to +latest value. Note this update is done in lieu of introducing another +set of visitors for each Java SE release.
+src/jdk.compiler/share/classes/com/sun/tools/javac/code/Source.java: +add anenumconstant for the new source version internal to +javac
+src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/ClassFile.java: +add anenumconstant for the new class file format version +internal tojavac
+src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/Target.java: +add anenumconstant for the new target version internal to +javac
+src/jdk.compiler/share/classes/com/sun/tools/javac/processing/PrintingProcessor.java+update printing processor to support the new source version
+- The symbol information for
--releaseis stored as new +text files in thesrc/jdk.compiler/share/data/symbols+directory, one file per module. The README file in that directory +contains directions on how to create the files.
+
+testfiles-
+
test/langtools/tools/javac/api/TestGetSourceVersions.java: +add newSourceVersionconstant to test matrix.
+test/langtools/tools/javac/classfiles/ClassVersionChecker.java: +add new enum constant for the new class file version
+test/langtools/tools/javac/lib/JavacTestingAbstractProcessor.java+update annotation processor extended byjavactests to +cover the new source version
+test/langtools/tools/javac/preview/classReaderTest/Client.nopreview.out+and +test/langtools/tools/javac/preview/classReaderTest/Client.preview.out: +update expected messages for preview errors and warnings
+
Testing the JDK
- Feature value of
- Non-US locale
- PKCS11 Tests +
Nearness
Prefer having checks inside test code.
Not only does having test logic outside, e.g. verification method, depending on asserts in product code contradict with several items above -but also decreases test’s readability and stability. It is much easier +but also decreases test's readability and stability. It is much easier to understand that a test is testing when all testing logic is located inside a test or nearby in shared test libraries. As a rule of thumb, the closer a check to a test, the better.
@@ -198,7 +198,7 @@Several checks
Prefer
EXPECToverASSERTif possible.This is related to the informativeness property of tests, information for other checks can help to better -localize a defect’s root-cause. One should use
@@ -235,7 +235,7 @@ASSERTif it +localize a defect's root-cause. One should useASSERTif it is impossible to continue test execution or if it does not make much sense. Later in the text,EXPECTforms will be used to refer to bothASSERT/EXPECT.Floating-point comparison
eps.C string comparison
Use string special macros for C strings comparisons.
-EXPECT_EQjust compares pointers’ values, which is +EXPECT_EQjust compares pointers' values, which is hardly what one wants comparing C strings. GoogleTest providesEXPECT_STREQandEXPECT_STRNEmacros to compare C string contents. There are also case-insensitive versions @@ -293,7 +293,7 @@Test group names
This naming scheme helps to find tests, filter them and simplifies test failure analysis. For example, class
Foo- test groupFoo, compiler logging subsystem - test group -CompilerLogging, G1 GC — test groupG1GC, and +CompilerLogging, G1 GC - test groupG1GC, and so forth.Filename
A test file must have
test_prefix and.cpp@@ -345,7 +345,7 @@Fixture classes
Friend classes
All test purpose friends should have either
-TestorTestablesuffix.It greatly simplifies understanding of friendship’s purpose and +
It greatly simplifies understanding of friendship's purpose and allows statically check that private members are not exposed unexpectedly. Having
FooTestas a friend ofFoowithout any comments will be understood as a necessary @@ -435,7 +435,7 @@Test-specific flags
Flag restoring
Restore changed flags.
It is quite common for tests to configure JVM in a certain way -changing flags’ values. GoogleTest provides two ways to set up +changing flags' values. GoogleTest provides two ways to set up environment before a test and restore it afterward: using either constructor and destructor or
SetUpandTearDownfunctions. Both ways require to use a test fixture @@ -444,7 +444,7 @@Flag restoring
be used in such cases to restore/set values.Caveats:
-
-
+ - ### Testing Ahead-of-time
+Optimizations
+
- Testing with alternative security providers @@ -411,6 +416,13 @@
JCOV
special targetjcov-testinstead oftest, e.g.make jcov-test TEST=jdk_lang. This will make sure the JCov image is built, and that JCov reporting is enabled. +To include JCov coverage for just a subset of all modules, you can +use the
+--with-jcov-modulesarguments to +configure, e.g. +--with-jcov-modules=jdk.compiler,java.desktop.For more fine-grained control, you can pass arbitrary filters to JCov +using
--with-jcov-filters, and you can specify a specific +JDK to instrument using--with-jcov-input-jdk.The JCov report is stored in
build/$BUILD/test-results/jcov-output/report.Please note that running with JCov reporting can be very memory @@ -589,6 +601,37 @@
PKCS11 Tests
JTREG="JAVA_OPTIONS=-Djdk.test.lib.artifacts.nsslib-linux_aarch64=/path/to/NSS-libs"For more notes about the PKCS11 tests, please refer to test/jdk/sun/security/pkcs11/README.
+### Testing Ahead-of-time +Optimizations
+One way to improve test coverage of ahead-of-time (AOT) optimizations +in the JDK is to run existing jtreg test cases in a special "AOT_JDK" +mode. Example:
+
+$ make test JTREG="AOT_JDK=onestep" \ + TEST=open/test/hotspot/jtreg/runtime/invokedynamicIn this testing mode, we first perform an AOT training run (see +https://openjdk.org/jeps/483) of a special test program (test/setup_aot/TestSetupAOT.java) +that accesses about 5,0000 classes in the JDK core libraries. +Optimization artifacts for these classes (such as pre-linked lambda +expressions, execution profiles, and pre-generated native code) are +stored into an AOT cache file, which will be used by all the JVMs +launched by the selected jtreg test cases.
+When the jtreg tests call into the core libraries classes that are in +the AOT cache, we will be able to test the AOT optimizations that were +used on those classes.
+Please note that not all existing jtreg test cases can be executed +with the AOT_JDK mode. See test/hotspot/jtreg/ProblemList-AotJdk.txt +and test/jdk/ProblemList-AotJdk.txt.
+Also, test cases that were written specifically to test AOT, such as +the tests under test/hotspot/jtreg/runtime/cds, +cannot be executed with the AOT_JDK mode.
+Valid values for
AOT_JDKareonestepand +twostep. These control how the AOT cache is generated. See +https://openjdk.org/jeps/514 for details. All other values are +ignored.Testing with alternative security providers
Some security tests use a hardcoded provider for diff --git a/doc/testing.md b/doc/testing.md index 351690c5e60..bb56c05c295 100644 --- a/doc/testing.md +++ b/doc/testing.md @@ -345,6 +345,14 @@ The simplest way to run tests with JCov coverage report is to use the special target `jcov-test` instead of `test`, e.g. `make jcov-test TEST=jdk_lang`. This will make sure the JCov image is built, and that JCov reporting is enabled. +To include JCov coverage for just a subset of all modules, you can use the +`--with-jcov-modules` arguments to `configure`, e.g. +`--with-jcov-modules=jdk.compiler,java.desktop`. + +For more fine-grained control, you can pass arbitrary filters to JCov using +`--with-jcov-filters`, and you can specify a specific JDK to instrument +using `--with-jcov-input-jdk`. + The JCov report is stored in `build/$BUILD/test-results/jcov-output/report`. Please note that running with JCov reporting can be very memory intensive. @@ -603,6 +611,43 @@ $ make test TEST="jtreg:sun/security/pkcs11/Secmod/AddTrustedCert.java" \ For more notes about the PKCS11 tests, please refer to test/jdk/sun/security/pkcs11/README. +### Testing Ahead-of-time Optimizations +------------------------------------------------------------------------------- +One way to improve test coverage of ahead-of-time (AOT) optimizations in +the JDK is to run existing jtreg test cases in a special "AOT_JDK" mode. +Example: + +``` +$ make test JTREG="AOT_JDK=onestep" \ + TEST=open/test/hotspot/jtreg/runtime/invokedynamic +``` + +In this testing mode, we first perform an AOT training run +(see https://openjdk.org/jeps/483) of a special test program +([test/setup_aot/TestSetupAOT.java](../test/setup_aot/TestSetupAOT.java)) +that accesses about 5,0000 classes in the JDK core libraries. +Optimization artifacts for these classes (such as pre-linked +lambda expressions, execution profiles, and pre-generated native code) +are stored into an AOT cache file, which will be used by all the JVMs +launched by the selected jtreg test cases. + +When the jtreg tests call into the core libraries classes that are in +the AOT cache, we will be able to test the AOT optimizations that were +used on those classes. + +Please note that not all existing jtreg test cases can be executed with +the AOT_JDK mode. See +[test/hotspot/jtreg/ProblemList-AotJdk.txt](../test/hotspot/jtreg/ProblemList-AotJdk.txt) +and [test/jdk/ProblemList-AotJdk.txt](../test/jdk/ProblemList-AotJdk.txt). + +Also, test cases that were written specifically to test AOT, such as the tests +under [test/hotspot/jtreg/runtime/cds](../test/hotspot/jtreg/runtime/cds/), +cannot be executed with the AOT_JDK mode. + +Valid values for `AOT_JDK` are `onestep` and `twostep`. These control how +the AOT cache is generated. See https://openjdk.org/jeps/514 for details. +All other values are ignored. + ### Testing with alternative security providers Some security tests use a hardcoded provider for `KeyFactory`, `Cipher`, diff --git a/make/Bundles.gmk b/make/Bundles.gmk index 8962b596278..ba8ec0c864b 100644 --- a/make/Bundles.gmk +++ b/make/Bundles.gmk @@ -174,9 +174,11 @@ else JRE_IMAGE_HOMEDIR := $(JRE_IMAGE_DIR) JDK_BUNDLE_SUBDIR := jdk-$(VERSION_NUMBER) JRE_BUNDLE_SUBDIR := jre-$(VERSION_NUMBER) + STATIC_JDK_BUNDLE_SUBDIR := static-jdk-$(VERSION_NUMBER) ifneq ($(DEBUG_LEVEL), release) JDK_BUNDLE_SUBDIR := $(JDK_BUNDLE_SUBDIR)/$(DEBUG_LEVEL) JRE_BUNDLE_SUBDIR := $(JRE_BUNDLE_SUBDIR)/$(DEBUG_LEVEL) + STATIC_JDK_BUNDLE_SUBDIR := $(STATIC_JDK_BUNDLE_SUBDIR)/$(DEBUG_LEVEL) endif # In certain situations, the JDK_IMAGE_DIR points to an image without the # the symbols and demos. If so, the symobls and demos can be found in a @@ -500,6 +502,21 @@ ifneq ($(filter static-libs-graal-bundles, $(MAKECMDGOALS)), ) STATIC_LIBS_GRAAL_TARGETS += $(BUILD_STATIC_LIBS_GRAAL_BUNDLE) endif +################################################################################# + +ifneq ($(filter static-jdk-bundles, $(MAKECMDGOALS)), ) + STATIC_JDK_BUNDLE_FILES := $(call FindFiles, $(STATIC_JDK_IMAGE_DIR)) + + $(eval $(call SetupBundleFile, BUILD_STATIC_JDK_BUNDLE, \ + BUNDLE_NAME := $(STATIC_JDK_BUNDLE_NAME), \ + FILES := $(STATIC_JDK_BUNDLE_FILES), \ + BASE_DIRS := $(STATIC_JDK_IMAGE_DIR), \ + SUBDIR := $(STATIC_JDK_BUNDLE_SUBDIR), \ + )) + + STATIC_JDK_TARGETS += $(BUILD_STATIC_JDK_BUNDLE) +endif + ################################################################################ product-bundles: $(PRODUCT_TARGETS) @@ -510,11 +527,12 @@ docs-javase-bundles: $(DOCS_JAVASE_TARGETS) docs-reference-bundles: $(DOCS_REFERENCE_TARGETS) static-libs-bundles: $(STATIC_LIBS_TARGETS) static-libs-graal-bundles: $(STATIC_LIBS_GRAAL_TARGETS) +static-jdk-bundles: $(STATIC_JDK_TARGETS) jcov-bundles: $(JCOV_TARGETS) .PHONY: product-bundles test-bundles \ docs-jdk-bundles docs-javase-bundles docs-reference-bundles \ - static-libs-bundles static-libs-graal-bundles jcov-bundles + static-libs-bundles static-libs-graal-bundles static-jdk-bundles jcov-bundles ################################################################################ diff --git a/make/CompileInterimLangtools.gmk b/make/CompileInterimLangtools.gmk index c869ea160c7..c7d1c3796f6 100644 --- a/make/CompileInterimLangtools.gmk +++ b/make/CompileInterimLangtools.gmk @@ -95,14 +95,16 @@ define SetupInterimModule SRC := $(BUILDTOOLS_OUTPUTDIR)/gensrc/$1.interim \ $$(wildcard $(SUPPORT_OUTPUTDIR)/gensrc/$1) \ $(TOPDIR)/src/$1/share/classes, \ - EXCLUDES := sun javax/tools/snippet-files, \ + EXCLUDES := sun, \ EXCLUDE_FILES := $(TOPDIR)/src/$1/share/classes/module-info.java \ $(TOPDIR)/src/$1/share/classes/javax/tools/ToolProvider.java \ $(TOPDIR)/src/$1/share/classes/com/sun/tools/javac/launcher/Main.java \ + $(TOPDIR)/src/$1/share/classes/com/sun/tools/javac/launcher/MemoryClassLoader.java \ $(TOPDIR)/src/$1/share/classes/com/sun/tools/javac/launcher/MemoryContext.java \ $(TOPDIR)/src/$1/share/classes/com/sun/tools/javac/launcher/MemoryModuleFinder.java \ $(TOPDIR)/src/$1/share/classes/com/sun/tools/javac/launcher/SourceLauncher.java \ Standard.java, \ + EXCLUDE_PATTERNS := -files, \ EXTRA_FILES := $(BUILDTOOLS_OUTPUTDIR)/gensrc/$1.interim/module-info.java \ $($1.interim_EXTRA_FILES), \ COPY := .gif .png .xml .css .svg .js .js.template .txt .woff .woff2 javax.tools.JavaCompilerTool, \ diff --git a/make/CompileJavaModules.gmk b/make/CompileJavaModules.gmk index b4a193dfade..1e26fb2b529 100644 --- a/make/CompileJavaModules.gmk +++ b/make/CompileJavaModules.gmk @@ -113,6 +113,7 @@ $(eval $(call SetupJavaCompilation, $(MODULE), \ DISABLED_WARNINGS := $(DISABLED_WARNINGS_java), \ EXCLUDES := $(EXCLUDES), \ EXCLUDE_FILES := $(EXCLUDE_FILES), \ + EXCLUDE_PATTERNS := -files, \ KEEP_ALL_TRANSLATIONS := $(KEEP_ALL_TRANSLATIONS), \ JAVAC_FLAGS := \ $(DOCLINT) \ diff --git a/make/Coverage.gmk b/make/Coverage.gmk index 2fd4e4ec6d4..a375c343185 100644 --- a/make/Coverage.gmk +++ b/make/Coverage.gmk @@ -34,21 +34,28 @@ else JCOV_INPUT_IMAGE_DIR := $(JDK_IMAGE_DIR) endif +JCOV_SUPPORT_DIR := $(SUPPORT_OUTPUTDIR)/jcov + #moving instrumented jdk image in and out of jcov_temp because of CODETOOLS-7902299 -JCOV_TEMP := $(SUPPORT_OUTPUTDIR)/jcov_temp +JCOV_TEMP := $(JCOV_SUPPORT_DIR)/temp + +ifneq ($(JCOV_MODULES), ) + JCOV_MODULES_FILTER := $(foreach m, $(JCOV_MODULES), -include_module $m) +endif $(JCOV_IMAGE_DIR)/release: $(JCOV_INPUT_IMAGE_DIR)/release $(call LogWarn, Creating instrumented jdk image with JCov) $(call MakeDir, $(JCOV_TEMP) $(IMAGES_OUTPUTDIR)) $(RM) -r $(JCOV_IMAGE_DIR) $(JCOV_TEMP)/* $(CP) -r $(JCOV_INPUT_IMAGE_DIR) $(JCOV_TEMP)/$(JCOV_IMAGE_SUBDIR) - $(JAVA) -Xmx3g -jar $(JCOV_HOME)/lib/jcov.jar JREInstr \ + $(call ExecuteWithLog, $(JCOV_SUPPORT_DIR)/run-jcov, \ + $(JAVA) -Xmx3g -jar $(JCOV_HOME)/lib/jcov.jar JREInstr \ -t $(JCOV_TEMP)/$(JCOV_IMAGE_SUBDIR)/template.xml \ -rt $(JCOV_HOME)/lib/jcov_network_saver.jar \ -exclude 'java.lang.Object' \ -exclude jdk.test.Main -exclude '**\$Proxy*' \ - $(JCOV_FILTERS) \ - $(JCOV_TEMP)/$(JCOV_IMAGE_SUBDIR) + $(JCOV_MODULES_FILTER) $(JCOV_FILTERS) \ + $(JCOV_TEMP)/$(JCOV_IMAGE_SUBDIR)) $(MV) $(JCOV_TEMP)/$(JCOV_IMAGE_SUBDIR) $(JCOV_IMAGE_DIR) $(RMDIR) $(JCOV_TEMP) diff --git a/make/Docs.gmk b/make/Docs.gmk index 60c029ce8f9..1fcc8575d2c 100644 --- a/make/Docs.gmk +++ b/make/Docs.gmk @@ -79,8 +79,6 @@ JAVADOC_TAGS := \ -tag see \ -taglet build.tools.taglet.ExtLink \ -taglet build.tools.taglet.Incubating \ - -taglet build.tools.taglet.PreviewNote \ - --preview-note-tag previewNote \ -tagletpath $(BUILDTOOLS_OUTPUTDIR)/jdk_tools_classes \ $(CUSTOM_JAVADOC_TAGS) \ # @@ -96,14 +94,14 @@ JAVADOC_DISABLED_DOCLINT_PACKAGES := org.w3c.* javax.smartcardio # The initial set of options for javadoc JAVADOC_OPTIONS := -use -keywords -notimestamp \ - -serialwarn -encoding ISO-8859-1 -docencoding UTF-8 -breakiterator \ + -serialwarn -encoding utf-8 -docencoding utf-8 -breakiterator \ -splitIndex --system none -javafx --expand-requires transitive \ --override-methods=summary # The reference options must stay stable to allow for comparisons across the # development cycle. REFERENCE_OPTIONS := -XDignore.symbol.file=true -use -keywords -notimestamp \ - -serialwarn -encoding ISO-8859-1 -breakiterator -splitIndex --system none \ + -serialwarn -encoding utf-8 -breakiterator -splitIndex --system none \ -html5 -javafx --expand-requires transitive # Should we add DRAFT stamps to the generated javadoc? @@ -262,7 +260,7 @@ define create_overview_file $$($1_OVERVIEW): $$($1_OVERVIEW_VARDEPS_FILE) $$(call LogInfo, Creating overview.html for $1) $$(call MakeDir, $$(@D)) - $$(ECHO) -n '$$($1_OVERVIEW_TEXT)' > $$@ + $$(PRINTF) "%s" '$$($1_OVERVIEW_TEXT)' > $$@ endef ################################################################################ @@ -542,7 +540,9 @@ $(eval $(call SetupApiDocsGeneration, REFERENCE_API, \ # Format: space-delimited list of names, including at most one '%' as a # wildcard. Spec source files match if their filename or any enclosing folder # name matches one of the items in SPEC_FILTER. -SPEC_FILTER := % +ifeq ($(SPEC_FILTER), ) + SPEC_FILTER := % +endif ApplySpecFilter = \ $(strip $(foreach file, $(1), \ diff --git a/make/GenerateLinkOptData.gmk b/make/GenerateLinkOptData.gmk index 5fc745ba223..6f6e1f29b4c 100644 --- a/make/GenerateLinkOptData.gmk +++ b/make/GenerateLinkOptData.gmk @@ -76,10 +76,14 @@ $(CLASSLIST_FILE): $(INTERIM_IMAGE_DIR)/bin/java$(EXECUTABLE_SUFFIX) $(CLASSLIST $(call LogInfo, Generating $(patsubst $(OUTPUTDIR)/%, %, $(JLI_TRACE_FILE))) $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java -XX:DumpLoadedClassList=$@.raw \ $(CLASSLIST_FILE_VM_OPTS) \ + -Xlog:aot=off \ + -Xlog:cds=off \ -cp $(SUPPORT_OUTPUTDIR)/classlist.jar \ build.tools.classlist.HelloClasslist $(LOG_DEBUG) $(GREP) -v HelloClasslist $@.raw > $@.interim $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java -Xshare:dump \ + -Xlog:aot=off \ + -Xlog:cds=off \ -XX:SharedClassListFile=$@.interim -XX:SharedArchiveFile=$@.jsa \ -Xmx128M -Xms128M $(LOG_INFO) $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java -XX:DumpLoadedClassList=$@.raw.2 \ @@ -87,6 +91,8 @@ $(CLASSLIST_FILE): $(INTERIM_IMAGE_DIR)/bin/java$(EXECUTABLE_SUFFIX) $(CLASSLIST -Djava.lang.invoke.MethodHandle.TRACE_RESOLVE=true \ $(CLASSLIST_FILE_VM_OPTS) \ --module-path $(SUPPORT_OUTPUTDIR)/classlist.jar \ + -Xlog:aot=off \ + -Xlog:cds=off \ -cp $(SUPPORT_OUTPUTDIR)/classlist.jar \ build.tools.classlist.HelloClasslist \ 2> $(LINK_OPT_DIR)/stderr > $(JLI_TRACE_FILE) \ @@ -100,6 +106,8 @@ $(CLASSLIST_FILE): $(INTERIM_IMAGE_DIR)/bin/java$(EXECUTABLE_SUFFIX) $(CLASSLIST $(GREP) -v HelloClasslist $@.raw.2 > $@.raw.3 $(GREP) -v @cp $@.raw.3 > $@.raw.4 $(FIXPATH) $(INTERIM_IMAGE_DIR)/bin/java \ + -Xlog:aot=off \ + -Xlog:cds=off \ -cp $(SUPPORT_OUTPUTDIR)/classlist.jar \ build.tools.classlist.SortClasslist $@.raw.4 > $@ diff --git a/make/Images.gmk b/make/Images.gmk index 3f2b34bc9c6..22e3e43cb1f 100644 --- a/make/Images.gmk +++ b/make/Images.gmk @@ -162,7 +162,7 @@ define CreateCDSArchive endif ifeq ($(DEBUG_CDS_ARCHIVE), true) - $1_$2_CDS_DUMP_FLAGS += -Xlog:cds+map*=trace:file=$$(JDK_IMAGE_DIR)/$$($1_$2_CDS_ARCHIVE).cdsmap:none:filesize=0 + $1_$2_CDS_DUMP_FLAGS += -Xlog:aot+map*=trace:file=$$(JDK_IMAGE_DIR)/$$($1_$2_CDS_ARCHIVE).cdsmap:none:filesize=0 endif $$(eval $$(call SetupExecute, $1_$2_gen_cds_archive_jdk, \ diff --git a/make/Init.gmk b/make/Init.gmk index 5dd1a71dd9a..38959323628 100644 --- a/make/Init.gmk +++ b/make/Init.gmk @@ -37,10 +37,6 @@ include MakeFileStart.gmk include $(TOPDIR)/make/InitSupport.gmk include LogUtils.gmk -# Force early generation of module-deps.gmk -GENERATE_MODULE_DEPS_FILE := true -include Modules.gmk - # Inclusion of this pseudo-target will cause make to execute this file # serially, regardless of -j. .NOTPARALLEL: @@ -114,7 +110,18 @@ reconfigure: CUSTOM_CONFIG_DIR="$(CUSTOM_CONFIG_DIR)" \ $(RECONFIGURE_COMMAND) ) -.PHONY: print-modules print-targets print-tests print-configuration reconfigure +# Create files that are needed to run most targets in Main.gmk +create-make-helpers: + ( cd $(TOPDIR) && \ + $(MAKE) $(MAKE_ARGS) -j 1 -f make/GenerateFindTests.gmk \ + $(USER_MAKE_VARS) ) + ( cd $(TOPDIR) && \ + $(MAKE) $(MAKE_ARGS) -j 1 -f make/Main.gmk $(USER_MAKE_VARS) \ + UPDATE_MODULE_DEPS=true NO_RECIPES=true \ + create-main-targets-include ) + +.PHONY: print-modules print-targets print-tests print-configuration \ + reconfigure create-make-helpers ############################################################################## # The main target. This will delegate all other targets into Main.gmk. @@ -134,7 +141,7 @@ TARGET_DESCRIPTION := target$(if $(word 2, $(MAIN_TARGETS)),s) \ # variables are explicitly propagated using $(USER_MAKE_VARS). main: MAKEOVERRIDES := -main: $(INIT_TARGETS) +main: $(INIT_TARGETS) create-make-helpers ifneq ($(SEQUENTIAL_TARGETS)$(PARALLEL_TARGETS), ) $(call RotateLogFiles) $(ECHO) "Building $(TARGET_DESCRIPTION)" $(BUILD_LOG_PIPE_SIMPLE) @@ -144,6 +151,9 @@ main: $(INIT_TARGETS) ( cd $(TOPDIR) && \ $(MAKE) $(MAKE_ARGS) -j 1 -f make/Main.gmk $(USER_MAKE_VARS) \ $(SEQUENTIAL_TARGETS) ) + # We might have cleaned away essential files, recreate them. + ( cd $(TOPDIR) && \ + $(MAKE) $(MAKE_ARGS) -j 1 -f make/Init.gmk create-make-helpers ) endif ifneq ($(PARALLEL_TARGETS), ) $(call PrepareFailureLogs) diff --git a/make/Main.gmk b/make/Main.gmk index eda3b79265a..d0568509a4e 100644 --- a/make/Main.gmk +++ b/make/Main.gmk @@ -42,6 +42,12 @@ include MakeFileStart.gmk include $(TOPDIR)/make/MainSupport.gmk include FindTests.gmk + +ifeq ($(UPDATE_MODULE_DEPS), true) + # Update module-deps.gmk if requested. This is read in Modules.gmk. + GENERATE_MODULE_DEPS_FILE := true +endif + include Modules.gmk # Are we requested to ignore dependencies? @@ -411,12 +417,14 @@ $(eval $(call SetupTarget, create-source-revision-tracker, \ )) BOOTCYCLE_TARGET := product-images +BOOTCYCLE_SPEC := $(dir $(SPEC))bootcycle-spec.gmk + bootcycle-images: ifneq ($(COMPILE_TYPE), cross) $(call LogWarn, Boot cycle build step 2: Building a new JDK image using previously built image) $(call MakeDir, $(OUTPUTDIR)/bootcycle-build) +$(MAKE) $(MAKE_ARGS) -f $(TOPDIR)/make/Init.gmk PARALLEL_TARGETS=$(BOOTCYCLE_TARGET) \ - LOG_PREFIX="[bootcycle] " JOBS= SPEC=$(dir $(SPEC))bootcycle-spec.gmk main + LOG_PREFIX="[bootcycle] " JOBS= SPEC=$(BOOTCYCLE_SPEC) main else $(call LogWarn, Boot cycle build disabled when cross compiling) endif @@ -875,6 +883,12 @@ $(eval $(call SetupTarget, static-libs-graal-bundles, \ DEPS := static-libs-graal-image, \ )) +$(eval $(call SetupTarget, static-jdk-bundles, \ + MAKEFILE := Bundles, \ + TARGET := static-jdk-bundles, \ + DEPS := static-jdk-image, \ +)) + ifeq ($(JCOV_ENABLED), true) $(eval $(call SetupTarget, jcov-bundles, \ MAKEFILE := Bundles, \ diff --git a/make/MainSupport.gmk b/make/MainSupport.gmk index ae4858c35af..d8dc894c1e9 100644 --- a/make/MainSupport.gmk +++ b/make/MainSupport.gmk @@ -57,7 +57,7 @@ define SetupTargetBody endef define CleanDocs - @$(ECHO) -n "Cleaning docs ..." + @$(PRINTF) "Cleaning docs ..." @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(SUPPORT_OUTPUTDIR)/docs $(RM) -r $(SUPPORT_OUTPUTDIR)/javadoc @@ -67,28 +67,28 @@ endef # Cleans the dir given as $1 define CleanDir - @$(ECHO) -n "Cleaning $(strip $1) build artifacts ..." + @$(PRINTF) "Cleaning %s build artifacts ..." "$(strip $1)" @$(ECHO) "" $(LOG_DEBUG) ($(CD) $(OUTPUTDIR) && $(RM) -r $1) @$(ECHO) " done" endef define CleanSupportDir - @$(ECHO) -n "Cleaning$(strip $1) build artifacts ..." + @$(PRINTF) "Cleaning %s build artifacts ..." "$(strip $1)" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(SUPPORT_OUTPUTDIR)/$(strip $1) @$(ECHO) " done" endef define CleanMakeSupportDir - @$(ECHO) -n "Cleaning $(strip $1) make support artifacts ..." + @$(PRINTF) "Cleaning %s make support artifacts ..." "$(strip $1)" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(MAKESUPPORT_OUTPUTDIR)/$(strip $1) @$(ECHO) " done" endef define CleanTest - @$(ECHO) -n "Cleaning test $(strip $1) ..." + @$(PRINTF) "Cleaning test %s ..." "$(strip $1)" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(SUPPORT_OUTPUTDIR)/test/$(strip $(subst -,/,$1)) # Remove as much of the test directory structure as is empty @@ -97,25 +97,25 @@ define CleanTest endef define Clean-gensrc - @$(ECHO) -n "Cleaning gensrc $(if $1,for $(strip $1) )..." + @$(PRINTF) "Cleaning gensrc %s..." "$(if $1,for $(strip $1) )" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(SUPPORT_OUTPUTDIR)/gensrc/$(strip $1) @$(ECHO) " done" endef define Clean-java - @$(ECHO) -n "Cleaning java $(if $1,for $(strip $1) )..." + @$(PRINTF) "Cleaning java %s..." "$(if $1,for $(strip $1) )" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(JDK_OUTPUTDIR)/modules/$(strip $1) $(RM) -r $(SUPPORT_OUTPUTDIR)/special_classes/$(strip $1) $(ECHO) " done" - $(ECHO) -n "Cleaning headers $(if $1,for $(strip $1) )..." + $(PRINTF) "Cleaning headers %s..." "$(if $1,for $(strip $1) )" $(RM) -r $(SUPPORT_OUTPUTDIR)/headers/$(strip $1) @$(ECHO) " done" endef define Clean-native - @$(ECHO) -n "Cleaning native $(if $1,for $(strip $1) )..." + @$(PRINTF) "Cleaning native %s..." "$(if $1,for $(strip $1) )" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(SUPPORT_OUTPUTDIR)/native/$(strip $1) $(RM) -r $(SUPPORT_OUTPUTDIR)/modules_libs/$(strip $1) @@ -124,7 +124,7 @@ define Clean-native endef define Clean-include - @$(ECHO) -n "Cleaning include $(if $1,for $(strip $1) )..." + @$(PRINTF) "Cleaning include %s..." "$(if $1,for $(strip $1) )" @$(ECHO) "" $(LOG_DEBUG) $(RM) -r $(SUPPORT_OUTPUTDIR)/modules_include/$(strip $1) @$(ECHO) " done" diff --git a/make/PreInit.gmk b/make/PreInit.gmk index a01971e845d..3df44308dd9 100644 --- a/make/PreInit.gmk +++ b/make/PreInit.gmk @@ -50,7 +50,8 @@ include $(TOPDIR)/make/Global.gmk # Targets provided by Init.gmk. ALL_INIT_TARGETS := print-modules print-targets print-configuration \ - print-tests reconfigure pre-compare-build post-compare-build + print-tests reconfigure pre-compare-build post-compare-build \ + create-make-helpers # CALLED_TARGETS is the list of targets that the user provided, # or "default" if unspecified. @@ -161,19 +162,19 @@ ifneq ($(SKIP_SPEC), true) ( cd $(TOPDIR) && \ $(foreach spec, $(SPECS), \ $(MAKE) $(MAKE_INIT_ARGS) -j 1 -f $(TOPDIR)/make/Init.gmk \ - SPEC=$(spec) $(MAKE_INIT_MAIN_TARGET_ARGS) \ - main && \ + SPEC=$(spec) TOPDIR_ALT=$(TOPDIR) \ + $(MAKE_INIT_MAIN_TARGET_ARGS) main && \ $(if $(and $(COMPARE_BUILD), $(PARALLEL_TARGETS)), \ $(MAKE) $(MAKE_INIT_ARGS) -f $(TOPDIR)/make/Init.gmk \ - SPEC=$(spec) \ + SPEC=$(spec) TOPDIR_ALT=$(TOPDIR) \ COMPARE_BUILD="$(COMPARE_BUILD)" \ pre-compare-build && \ $(MAKE) $(MAKE_INIT_ARGS) -j 1 -f $(TOPDIR)/make/Init.gmk \ - SPEC=$(spec) $(MAKE_INIT_MAIN_TARGET_ARGS) \ + SPEC=$(spec) TOPDIR_ALT=$(TOPDIR) \ COMPARE_BUILD="$(COMPARE_BUILD):NODRYRUN=true" \ - main && \ + $(MAKE_INIT_MAIN_TARGET_ARGS) main && \ $(MAKE) $(MAKE_INIT_ARGS) -f $(TOPDIR)/make/Init.gmk \ - SPEC=$(spec) \ + SPEC=$(spec) TOPDIR_ALT=$(TOPDIR) \ COMPARE_BUILD="$(COMPARE_BUILD):NODRYRUN=true" \ post-compare-build && \ ) \ diff --git a/make/PreInitSupport.gmk b/make/PreInitSupport.gmk index 668363d8725..660e1214b5b 100644 --- a/make/PreInitSupport.gmk +++ b/make/PreInitSupport.gmk @@ -250,13 +250,14 @@ endef # Param 1: FORCE = force generation of main-targets.gmk or LAZY = do not force. # Param 2: The SPEC file to use. define DefineMainTargets + SPEC_FILE := $(strip $2) # We will start by making sure the main-targets.gmk file is removed, if # make has not been restarted. By the -include, we will trigger the # rule for generating the file (which is never there since we removed it), # thus generating it fresh, and make will restart, incrementing the restart # count. - main_targets_file := $$(dir $(strip $2))make-support/main-targets.gmk + main_targets_file := $$(dir $$(SPEC_FILE))make-support/main-targets.gmk ifeq ($$(MAKE_RESTARTS), ) # Only do this if make has not been restarted, and if we do not force it. @@ -267,11 +268,12 @@ define DefineMainTargets $$(main_targets_file): @( cd $$(TOPDIR) && \ - $$(MAKE) $$(MAKE_LOG_FLAGS) -r -R -f $$(TOPDIR)/make/GenerateFindTests.gmk \ - -I $$(TOPDIR)/make/common SPEC=$(strip $2) ) + $$(MAKE) $$(MAKE_LOG_FLAGS) -s -r -R -f $$(TOPDIR)/make/GenerateFindTests.gmk \ + -I $$(TOPDIR)/make/common SPEC=$$(SPEC_FILE) TOPDIR_ALT=$$(TOPDIR)) @( cd $$(TOPDIR) && \ - $$(MAKE) $$(MAKE_LOG_FLAGS) -r -R -f $$(TOPDIR)/make/Main.gmk \ - -I $$(TOPDIR)/make/common SPEC=$(strip $2) NO_RECIPES=true \ + $$(MAKE) $$(MAKE_LOG_FLAGS) -s -r -R -f $$(TOPDIR)/make/Main.gmk \ + -I $$(TOPDIR)/make/common SPEC=$$(SPEC_FILE) TOPDIR_ALT=$$(TOPDIR) \ + UPDATE_MODULE_DEPS=true NO_RECIPES=true \ $$(MAKE_LOG_VARS) \ create-main-targets-include ) diff --git a/make/RunTests.gmk b/make/RunTests.gmk index 80c1ff99b2e..60ae1bd4763 100644 --- a/make/RunTests.gmk +++ b/make/RunTests.gmk @@ -115,6 +115,7 @@ JTREG_COV_OPTIONS := ifeq ($(TEST_OPTS_JCOV), true) JCOV_OUTPUT_DIR := $(TEST_RESULTS_DIR)/jcov-output + JCOV_SUPPORT_DIR := $(TEST_SUPPORT_DIR)/jcov-support JCOV_GRABBER_LOG := $(JCOV_OUTPUT_DIR)/grabber.log JCOV_RESULT_FILE := $(JCOV_OUTPUT_DIR)/result.xml JCOV_REPORT := $(JCOV_OUTPUT_DIR)/report @@ -582,6 +583,8 @@ define SetMicroValue else ifneq ($3, ) $1_$2 := $3 + else + $1_$2 := endif endif endef @@ -708,6 +711,8 @@ define SetJtregValue else ifneq ($3, ) $1_$2 := $3 + else + $1_$2 := endif endif endif @@ -720,6 +725,7 @@ endef # Parameter 1 is the name of the rule. # # Remaining parameters are named arguments. +# TRAINING The AOT training mode: onestep or twostep # VM_OPTIONS List of JVM arguments to use when creating AOT cache # # After calling this, the following variables are defined @@ -748,23 +754,39 @@ define SetupAOTBody $$($1_AOT_JDK_CACHE): $$(JDK_IMAGE_DIR)/release $$(call MakeDir, $$($1_AOT_JDK_OUTPUT_DIR)) - $$(call LogWarn, AOT: Create cache configuration) \ - $$(call ExecuteWithLog, $$($1_AOT_JDK_OUTPUT_DIR), ( \ - cd $$($1_AOT_JDK_OUTPUT_DIR); \ - $(JAR) --extract --file $(TEST_IMAGE_DIR)/setup_aot/TestSetupAOT.jar; \ - $$(FIXPATH) $(JDK_UNDER_TEST)/bin/java $$($1_VM_OPTIONS) \ - -Xlog:class+load,cds,cds+class=debug:file=$$($1_AOT_JDK_CONF).log -Xlog:cds*=error \ - -XX:AOTMode=record -XX:AOTConfiguration=$$($1_AOT_JDK_CONF) \ - TestSetupAOT $$($1_AOT_JDK_OUTPUT_DIR) > $$($1_AOT_JDK_LOG) \ - )) - - $$(call LogWarn, AOT: Generate AOT cache $$($1_AOT_JDK_CACHE) with flags: $$($1_VM_OPTIONS)) - $$(call ExecuteWithLog, $$($1_AOT_JDK_OUTPUT_DIR), ( \ - $$(FIXPATH) $(JDK_UNDER_TEST)/bin/java \ - $$($1_VM_OPTIONS) -Xlog:cds,cds+class=debug:file=$$($1_AOT_JDK_CACHE).log -Xlog:cds*=error \ - -XX:ExtraSharedClassListFile=$(JDK_UNDER_TEST)/lib/classlist \ - -XX:AOTMode=create -XX:AOTConfiguration=$$($1_AOT_JDK_CONF) -XX:AOTCache=$$($1_AOT_JDK_CACHE) \ - )) + ifeq ($$($1_TRAINING), onestep) + + $$(call LogWarn, AOT: Create AOT cache $$($1_AOT_JDK_CACHE) in one step with flags: $$($1_VM_OPTIONS)) \ + $$(call ExecuteWithLog, $$($1_AOT_JDK_OUTPUT_DIR), ( \ + cd $$($1_AOT_JDK_OUTPUT_DIR); \ + $(JAR) --extract --file $(TEST_IMAGE_DIR)/setup_aot/TestSetupAOT.jar; \ + $$(FIXPATH) $(JDK_UNDER_TEST)/bin/java $$($1_VM_OPTIONS) \ + -Xlog:class+load,aot,aot+class=debug:file=$$($1_AOT_JDK_CACHE).log -Xlog:cds*=error -Xlog:aot*=error \ + -XX:AOTMode=record -XX:AOTCacheOutput=$$($1_AOT_JDK_CACHE) \ + TestSetupAOT $$($1_AOT_JDK_OUTPUT_DIR) > $$($1_AOT_JDK_LOG) \ + )) + + else + + $$(call LogWarn, AOT: Create cache configuration) \ + $$(call ExecuteWithLog, $$($1_AOT_JDK_OUTPUT_DIR), ( \ + cd $$($1_AOT_JDK_OUTPUT_DIR); \ + $(JAR) --extract --file $(TEST_IMAGE_DIR)/setup_aot/TestSetupAOT.jar; \ + $$(FIXPATH) $(JDK_UNDER_TEST)/bin/java $$($1_VM_OPTIONS) \ + -Xlog:class+load,aot,aot+class=debug:file=$$($1_AOT_JDK_CONF).log -Xlog:cds*=error -Xlog:aot*=error \ + -XX:AOTMode=record -XX:AOTConfiguration=$$($1_AOT_JDK_CONF) \ + TestSetupAOT $$($1_AOT_JDK_OUTPUT_DIR) > $$($1_AOT_JDK_LOG) \ + )) + + $$(call LogWarn, AOT: Generate AOT cache $$($1_AOT_JDK_CACHE) with flags: $$($1_VM_OPTIONS)) + $$(call ExecuteWithLog, $$($1_AOT_JDK_OUTPUT_DIR), ( \ + $$(FIXPATH) $(JDK_UNDER_TEST)/bin/java \ + $$($1_VM_OPTIONS) -Xlog:aot,aot+class=debug:file=$$($1_AOT_JDK_CACHE).log -Xlog:cds*=error -Xlog:aot*=error \ + -XX:ExtraSharedClassListFile=$(JDK_UNDER_TEST)/lib/classlist \ + -XX:AOTMode=create -XX:AOTConfiguration=$$($1_AOT_JDK_CONF) -XX:AOTCache=$$($1_AOT_JDK_CACHE) \ + )) + + endif $1_AOT_TARGETS += $$($1_AOT_JDK_CACHE) @@ -830,7 +852,7 @@ define SetupRunJtregTestBody JTREG_RETRY_COUNT ?= 0 JTREG_REPEAT_COUNT ?= 0 JTREG_REPORT ?= files - JTREG_AOT_JDK ?= false + JTREG_AOT_JDK ?= none ifneq ($$(JTREG_RETRY_COUNT), 0) ifneq ($$(JTREG_REPEAT_COUNT), 0) @@ -840,6 +862,12 @@ define SetupRunJtregTestBody endif endif + ifeq ($$(JTREG_RUN_PROBLEM_LISTS), true) + JTREG_PROBLEM_LIST_PREFIX := -match: + else + JTREG_PROBLEM_LIST_PREFIX := -exclude: + endif + ifneq ($$(JTREG_TEST_THREAD_FACTORY), ) $1_JTREG_BASIC_OPTIONS += -testThreadFactoryPath:$$(JTREG_TEST_THREAD_FACTORY_JAR) $1_JTREG_BASIC_OPTIONS += -testThreadFactory:$$(JTREG_TEST_THREAD_FACTORY) @@ -868,7 +896,7 @@ define SetupRunJtregTestBody # version of the JDK. $1_JTREG_BASIC_OPTIONS += -$$($1_JTREG_TEST_MODE) \ -verbose:$$(JTREG_VERBOSE) -retain:$$(JTREG_RETAIN) \ - -concurrency:$$($1_JTREG_JOBS) -timeoutFactor:$$(JTREG_TIMEOUT_FACTOR) \ + -concurrency:$$($1_JTREG_JOBS) \ -vmoption:-XX:MaxRAMPercentage=$$($1_JTREG_MAX_RAM_PERCENTAGE) \ -vmoption:-Dtest.boot.jdk="$$(BOOT_JDK)" \ -vmoption:-Djava.io.tmpdir="$$($1_TEST_TMP_DIR)" @@ -901,12 +929,6 @@ define SetupRunJtregTestBody $1_JTREG_BASIC_OPTIONS += -nativepath:$$($1_JTREG_NATIVEPATH) endif - ifeq ($$(JTREG_RUN_PROBLEM_LISTS), true) - JTREG_PROBLEM_LIST_PREFIX := -match: - else - JTREG_PROBLEM_LIST_PREFIX := -exclude: - endif - ifneq ($$($1_JTREG_PROBLEM_LIST), ) $1_JTREG_BASIC_OPTIONS += $$(addprefix $$(JTREG_PROBLEM_LIST_PREFIX), $$($1_JTREG_PROBLEM_LIST)) endif @@ -929,6 +951,11 @@ define SetupRunJtregTestBody JTREG_AUTO_PROBLEM_LISTS += ProblemList-shenandoah.txt endif + ifneq ($$(findstring --enable-preview, $$(JTREG_ALL_OPTIONS)), ) + JTREG_AUTO_PROBLEM_LISTS += ProblemList-enable-preview.txt + endif + + ifneq ($$(JTREG_EXTRA_PROBLEM_LISTS), ) # Accept both absolute paths as well as relative to the current test root. $1_JTREG_BASIC_OPTIONS += $$(addprefix $$(JTREG_PROBLEM_LIST_PREFIX), $$(wildcard \ @@ -965,12 +992,12 @@ define SetupRunJtregTestBody endif endif - ifeq ($$(JTREG_AOT_JDK), true) + ifneq ($$(filter $$(JTREG_AOT_JDK), onestep twostep), ) $$(call LogWarn, Add AOT target for $1) - $$(eval $$(call SetupAOT, $1, VM_OPTIONS := $$(JTREG_ALL_OPTIONS) )) - + $$(eval $$(call SetupAOT, $1, \ + TRAINING := $$(JTREG_AOT_JDK), \ + VM_OPTIONS := $$(JTREG_ALL_OPTIONS) )) $$(call LogWarn, AOT_JDK_CACHE=$$($1_AOT_JDK_CACHE)) - $1_JTREG_BASIC_OPTIONS += -vmoption:-XX:AOTCache="$$($1_AOT_JDK_CACHE)" endif @@ -988,6 +1015,7 @@ define SetupRunJtregTestBody endif JTREG_TIMEOUT_FACTOR ?= $$(JTREG_AUTO_TIMEOUT_FACTOR) + $1_JTREG_BASIC_OPTIONS += -timeoutFactor:$$(JTREG_TIMEOUT_FACTOR) clean-outputdirs-$1: $$(call LogWarn, Clean up dirs for $1) @@ -1338,12 +1366,14 @@ TARGETS += run-all-tests pre-run-test post-run-test run-test-report run-test ifeq ($(TEST_OPTS_JCOV), true) + JCOV_VM_OPTS := -Xmx4g -Djdk.xml.totalEntitySizeLimit=0 -Djdk.xml.maxGeneralEntitySizeLimit=0 + jcov-do-start-grabber: $(call MakeDir, $(JCOV_OUTPUT_DIR)) if $(JAVA) -jar $(JCOV_HOME)/lib/jcov.jar GrabberManager -status 1>/dev/null 2>&1 ; then \ $(JAVA) -jar $(JCOV_HOME)/lib/jcov.jar GrabberManager -stop -stoptimeout 3600 ; \ fi - $(JAVA) -Xmx4g -jar $(JCOV_HOME)/lib/jcov.jar Grabber -v -t \ + $(JAVA) $(JCOV_VM_OPTS) -jar $(JCOV_HOME)/lib/jcov.jar Grabber -v -t \ $(JCOV_IMAGE_DIR)/template.xml -o $(JCOV_RESULT_FILE) \ 1>$(JCOV_GRABBER_LOG) 2>&1 & @@ -1356,6 +1386,10 @@ ifeq ($(TEST_OPTS_JCOV), true) $(JAVA) -jar $(JCOV_HOME)/lib/jcov.jar GrabberManager -stop -stoptimeout 3600 JCOV_REPORT_TITLE := JDK code coverage report
+ ifneq ($(JCOV_MODULES), ) + JCOV_MODULES_FILTER := $(foreach m, $(JCOV_MODULES), -include_module $m) + JCOV_REPORT_TITLE += Included modules: $(JCOV_MODULES)
+ endif ifneq ($(JCOV_FILTERS), ) JCOV_REPORT_TITLE += Code filters: $(JCOV_FILTERS)
endif @@ -1363,11 +1397,12 @@ ifeq ($(TEST_OPTS_JCOV), true) jcov-gen-report: jcov-stop-grabber $(call LogWarn, Generating JCov report ...) - $(JAVA) -Xmx4g -jar $(JCOV_HOME)/lib/jcov.jar RepGen -sourcepath \ + $(call ExecuteWithLog, $(JCOV_SUPPORT_DIR)/run-jcov-repgen, \ + $(JAVA) $(JCOV_VM_OPTS) -jar $(JCOV_HOME)/lib/jcov.jar RepGen -sourcepath \ `$(ECHO) $(TOPDIR)/src/*/share/classes/ | $(TR) ' ' ':'` -fmt html \ - $(JCOV_FILTERS) \ + $(JCOV_MODULES_FILTER) $(JCOV_FILTERS) \ -mainReportTitle "$(JCOV_REPORT_TITLE)" \ - -o $(JCOV_REPORT) $(JCOV_RESULT_FILE) + -o $(JCOV_REPORT) $(JCOV_RESULT_FILE)) TARGETS += jcov-do-start-grabber jcov-start-grabber jcov-stop-grabber \ jcov-gen-report @@ -1387,7 +1422,7 @@ ifeq ($(TEST_OPTS_JCOV), true) jcov-gen-diffcoverage: jcov-stop-grabber $(call LogWarn, Generating diff coverage with changeset $(TEST_OPTS_JCOV_DIFF_CHANGESET) ... ) $(DIFF_COMMAND) - $(JAVA) -Xmx4g -jar $(JCOV_HOME)/lib/jcov.jar \ + $(JAVA) $(JCOV_VM_OPTS) -jar $(JCOV_HOME)/lib/jcov.jar \ DiffCoverage -replaceDiff "src/.*/classes/:" -all \ $(JCOV_RESULT_FILE) $(JCOV_SOURCE_DIFF) > \ $(JCOV_DIFF_COVERAGE_REPORT) diff --git a/make/RunTestsPrebuilt.gmk b/make/RunTestsPrebuilt.gmk index ba9731789b0..f5fe1d33830 100644 --- a/make/RunTestsPrebuilt.gmk +++ b/make/RunTestsPrebuilt.gmk @@ -217,9 +217,9 @@ else ifeq ($(OPENJDK_TARGET_OS), macosx) else ifeq ($(OPENJDK_TARGET_OS), windows) NUM_CORES := $(NUMBER_OF_PROCESSORS) MEMORY_SIZE := $(shell \ - $(EXPR) `wmic computersystem get totalphysicalmemory -value \ - | $(GREP) = | $(SED) 's/\\r//g' \ - | $(CUT) -d "=" -f 2-` / 1024 / 1024 \ + $(EXPR) `powershell -Command \ + "(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory" \ + | $(SED) 's/\\r//g' ` / 1024 / 1024 \ ) endif ifeq ($(NUM_CORES), ) diff --git a/make/autoconf/basic.m4 b/make/autoconf/basic.m4 index 6daba35547b..2d3e071dd52 100644 --- a/make/autoconf/basic.m4 +++ b/make/autoconf/basic.m4 @@ -134,17 +134,33 @@ AC_DEFUN_ONCE([BASIC_SETUP_BUILD_ENV], ) AC_SUBST(BUILD_ENV) + AC_MSG_CHECKING([for locale to use]) if test "x$LOCALE" != x; then # Check if we actually have C.UTF-8; if so, use it if $LOCALE -a | $GREP -q -E "^C\.(utf8|UTF-8)$"; then LOCALE_USED=C.UTF-8 + AC_MSG_RESULT([C.UTF-8 (recommended)]) + elif $LOCALE -a | $GREP -q -E "^en_US\.(utf8|UTF-8)$"; then + LOCALE_USED=en_US.UTF-8 + AC_MSG_RESULT([en_US.UTF-8 (acceptable fallback)]) else - AC_MSG_WARN([C.UTF-8 locale not found, using C locale]) - LOCALE_USED=C + # As a fallback, check if users locale is UTF-8. USER_LOCALE was saved + # by the wrapper configure script before autconf messed up LC_ALL. + if $ECHO $USER_LOCALE | $GREP -q -E "\.(utf8|UTF-8)$"; then + LOCALE_USED=$USER_LOCALE + AC_MSG_RESULT([$USER_LOCALE (untested fallback)]) + AC_MSG_WARN([Could not find C.UTF-8 or en_US.UTF-8 locale. This is not supported, and the build might fail unexpectedly.]) + else + AC_MSG_RESULT([no UTF-8 locale found]) + AC_MSG_WARN([No UTF-8 locale found. This is not supported. Proceeding with the C locale, but the build might fail unexpectedly.]) + LOCALE_USED=C + fi + AC_MSG_NOTICE([The recommended locale is C.UTF-8, but en_US.UTF-8 is also accepted.]) fi else - AC_MSG_WARN([locale command not not found, using C locale]) - LOCALE_USED=C + LOCALE_USED=C.UTF-8 + AC_MSG_RESULT([C.UTF-8 (default)]) + AC_MSG_WARN([locale command not not found, using C.UTF-8 locale]) fi export LC_ALL=$LOCALE_USED @@ -399,11 +415,21 @@ AC_DEFUN_ONCE([BASIC_SETUP_OUTPUT_DIR], [ CONF_NAME=${with_conf_name} ]) # Test from where we are running configure, in or outside of src root. + if test "x$OPENJDK_BUILD_OS" = xwindows || test "x$OPENJDK_BUILD_OS" = "xmacosx"; then + # These systems have case insensitive paths, so convert them to lower case. + [ cmp_configure_start_dir=`$ECHO $CONFIGURE_START_DIR | $TR '[:upper:]' '[:lower:]'` ] + [ cmp_topdir=`$ECHO $TOPDIR | $TR '[:upper:]' '[:lower:]'` ] + [ cmp_custom_root=`$ECHO $CUSTOM_ROOT | $TR '[:upper:]' '[:lower:]'` ] + else + cmp_configure_start_dir="$CONFIGURE_START_DIR" + cmp_topdir="$TOPDIR" + cmp_custom_root="$CUSTOM_ROOT" + fi AC_MSG_CHECKING([where to store configuration]) - if test "x$CONFIGURE_START_DIR" = "x$TOPDIR" \ - || test "x$CONFIGURE_START_DIR" = "x$CUSTOM_ROOT" \ - || test "x$CONFIGURE_START_DIR" = "x$TOPDIR/make/autoconf" \ - || test "x$CONFIGURE_START_DIR" = "x$TOPDIR/make" ; then + if test "x$cmp_configure_start_dir" = "x$cmp_topdir" \ + || test "x$cmp_configure_start_dir" = "x$cmp_custom_root" \ + || test "x$cmp_configure_start_dir" = "x$cmp_topdir/make/autoconf" \ + || test "x$cmp_configure_start_dir" = "x$cmp_topdir/make" ; then # We are running configure from the src root. # Create a default ./build/target-variant-debuglevel output root. if test "x${CONF_NAME}" = x; then @@ -424,7 +450,12 @@ AC_DEFUN_ONCE([BASIC_SETUP_OUTPUT_DIR], # If configuration is situated in normal build directory, just use the build # directory name as configuration name, otherwise use the complete path. if test "x${CONF_NAME}" = x; then - CONF_NAME=`$ECHO $CONFIGURE_START_DIR | $SED -e "s!^${TOPDIR}/build/!!"` + [ if [[ "$cmp_configure_start_dir" =~ ^${cmp_topdir}/build/[^/]+$ || + "$cmp_configure_start_dir" =~ ^${cmp_custom_root}/build/[^/]+$ ]]; then ] + CONF_NAME="${CONFIGURE_START_DIR##*/}" + else + CONF_NAME="$CONFIGURE_START_DIR" + fi fi OUTPUTDIR="$CONFIGURE_START_DIR" AC_MSG_RESULT([in current directory]) diff --git a/make/autoconf/basic_windows.m4 b/make/autoconf/basic_windows.m4 index fb6fc526bfa..dac6ec15db6 100644 --- a/make/autoconf/basic_windows.m4 +++ b/make/autoconf/basic_windows.m4 @@ -1,5 +1,5 @@ # -# Copyright (c) 2011, 2022, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2011, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -159,7 +159,7 @@ AC_DEFUN([BASIC_SETUP_PATHS_WINDOWS], else WINENV_PREFIX_ARG="$WINENV_PREFIX" fi - FIXPATH_ARGS="-e $PATHTOOL -p $WINENV_PREFIX_ARG -r ${WINENV_ROOT//\\/\\\\} -t $WINENV_TEMP_DIR -c $CMD -q" + FIXPATH_ARGS="-e $PATHTOOL -p $WINENV_PREFIX_ARG -r ${WINENV_ROOT//\\/\\\\} -t $WINENV_TEMP_DIR -c $CMD" FIXPATH_BASE="$BASH $FIXPATH_DIR/fixpath.sh $FIXPATH_ARGS" FIXPATH="$FIXPATH_BASE exec" @@ -215,7 +215,7 @@ AC_DEFUN([BASIC_WINDOWS_FINALIZE_FIXPATH], if test "x$OPENJDK_BUILD_OS" = xwindows; then FIXPATH_CMDLINE=". $TOPDIR/make/scripts/fixpath.sh -e $PATHTOOL \ -p $WINENV_PREFIX_ARG -r ${WINENV_ROOT//\\/\\\\} -t $WINENV_TEMP_DIR \ - -c $CMD -q" + -c $CMD" $ECHO > $OUTPUTDIR/fixpath '#!/bin/bash' $ECHO >> $OUTPUTDIR/fixpath export PATH='"[$]PATH:'$PATH'"' $ECHO >> $OUTPUTDIR/fixpath $FIXPATH_CMDLINE '"[$]@"' diff --git a/make/autoconf/boot-jdk.m4 b/make/autoconf/boot-jdk.m4 index d39e6e75a94..feb16c7d179 100644 --- a/make/autoconf/boot-jdk.m4 +++ b/make/autoconf/boot-jdk.m4 @@ -1,5 +1,5 @@ # -# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2011, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -180,11 +180,13 @@ AC_DEFUN([BOOTJDK_CHECK_JAVA_HOME], # Test: Is there a java or javac in the PATH, which is a symlink to the JDK? AC_DEFUN([BOOTJDK_CHECK_JAVA_IN_PATH_IS_SYMLINK], [ - UTIL_LOOKUP_PROGS(JAVAC_CHECK, javac, , NOFIXPATH) - UTIL_LOOKUP_PROGS(JAVA_CHECK, java, , NOFIXPATH) - BINARY="$JAVAC_CHECK" - if test "x$JAVAC_CHECK" = x; then - BINARY="$JAVA_CHECK" + UTIL_LOOKUP_PROGS(JAVAC_CHECK, javac) + UTIL_GET_EXECUTABLE(JAVAC_CHECK) # Will setup JAVAC_CHECK_EXECUTABLE + UTIL_LOOKUP_PROGS(JAVA_CHECK, java) + UTIL_GET_EXECUTABLE(JAVA_CHECK) # Will setup JAVA_CHECK_EXECUTABLE + BINARY="$JAVAC_CHECK_EXECUTABLE" + if test "x$JAVAC_CHECK_EXECUTABLE" = x; then + BINARY="$JAVA_CHECK_EXECUTABLE" fi if test "x$BINARY" != x; then # So there is a java(c) binary, it might be part of a JDK. diff --git a/make/autoconf/build-performance.m4 b/make/autoconf/build-performance.m4 index 4414ea0d93c..10e86e75199 100644 --- a/make/autoconf/build-performance.m4 +++ b/make/autoconf/build-performance.m4 @@ -1,5 +1,5 @@ # -# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2011, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -75,7 +75,8 @@ AC_DEFUN([BPERF_CHECK_MEMORY_SIZE], FOUND_MEM=yes elif test "x$OPENJDK_BUILD_OS" = xwindows; then # Windows, but without cygwin - MEMORY_SIZE=`wmic computersystem get totalphysicalmemory -value | grep = | cut -d "=" -f 2-` + MEMORY_SIZE=`powershell -Command \ + "(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory" | $SED 's/\\r//g' ` MEMORY_SIZE=`expr $MEMORY_SIZE / 1024 / 1024` FOUND_MEM=yes fi diff --git a/make/autoconf/configure b/make/autoconf/configure index 6fa0aacfbc9..443a37bae77 100644 --- a/make/autoconf/configure +++ b/make/autoconf/configure @@ -49,7 +49,9 @@ fi export CONFIG_SHELL=$BASH export _as_can_reexec=no -# Make sure all shell commands are executed with the C locale +# Save user's current locale, but make sure all future shell commands are +# executed with the C locale +export USER_LOCALE=$LC_ALL export LC_ALL=C if test "x$CUSTOM_CONFIG_DIR" != x; then diff --git a/make/autoconf/flags-cflags.m4 b/make/autoconf/flags-cflags.m4 index bafedddf04f..e80d9a98957 100644 --- a/make/autoconf/flags-cflags.m4 +++ b/make/autoconf/flags-cflags.m4 @@ -573,12 +573,20 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_HELPER], TOOLCHAIN_CFLAGS_JDK="$TOOLCHAIN_CFLAGS_JDK -fvisibility=hidden -fstack-protector" elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then - # The -utf-8 option sets source and execution character sets to UTF-8 to enable correct - # compilation of all source files regardless of the active code page on Windows. - TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:inline -Zc:throwingNew -permissive- -utf-8 -MP" - TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:inline -Zc:throwingNew -permissive- -utf-8 -Zc:wchar_t-" + TOOLCHAIN_CFLAGS_JVM="-nologo -MD -Zc:preprocessor -Zc:inline -Zc:throwingNew -permissive- -MP" + TOOLCHAIN_CFLAGS_JDK="-nologo -MD -Zc:preprocessor -Zc:inline -Zc:throwingNew -permissive- -Zc:wchar_t-" fi + # Set character encoding in source + if test "x$TOOLCHAIN_TYPE" = xgcc || test "x$TOOLCHAIN_TYPE" = xclang; then + CHARSET_CFLAGS="-finput-charset=utf-8" + elif test "x$TOOLCHAIN_TYPE" = xmicrosoft; then + # The -utf-8 option sets both source and execution character sets + CHARSET_CFLAGS="-utf-8 -validate-charset" + fi + TOOLCHAIN_CFLAGS_JVM="$TOOLCHAIN_CFLAGS_JVM $CHARSET_CFLAGS" + TOOLCHAIN_CFLAGS_JDK="$TOOLCHAIN_CFLAGS_JDK $CHARSET_CFLAGS" + # CFLAGS C language level for JDK sources (hotspot only uses C++) if test "x$TOOLCHAIN_TYPE" = xgcc || test "x$TOOLCHAIN_TYPE" = xclang; then LANGSTD_CFLAGS="-std=c11" @@ -724,8 +732,7 @@ AC_DEFUN([FLAGS_SETUP_CFLAGS_CPU_DEP], $1_CFLAGS_CPU_JVM="-mno-multiple -mno-string" if test "x$FLAGS_CPU" = xppc64; then # -mminimal-toc fixes `relocation truncated to fit' error for gcc 4.1. - # Use ppc64 instructions, but schedule for power5 - $1_CFLAGS_CPU="-mcpu=powerpc64 -mtune=power5" + $1_CFLAGS_CPU="-mcpu=power8 -mtune=power8" $1_CFLAGS_CPU_JVM="${$1_CFLAGS_CPU_JVM} -mminimal-toc" elif test "x$FLAGS_CPU" = xppc64le; then # Little endian machine uses ELFv2 ABI. diff --git a/make/autoconf/help.m4 b/make/autoconf/help.m4 index 93796d27f06..d8c0b2ffaef 100644 --- a/make/autoconf/help.m4 +++ b/make/autoconf/help.m4 @@ -292,12 +292,12 @@ AC_DEFUN_ONCE([HELP_PRINT_SUMMARY_AND_WARNINGS], $ECHO "* Debug level: $DEBUG_LEVEL" $ECHO "* HS debug level: $HOTSPOT_DEBUG_LEVEL" $ECHO "* JVM variants: $JVM_VARIANTS" - $ECHO -n "* JVM features: " + $PRINTF "* JVM features: " for variant in $JVM_VARIANTS; do features_var_name=JVM_FEATURES_$variant JVM_FEATURES_FOR_VARIANT=${!features_var_name} - $ECHO -n "$variant: '$JVM_FEATURES_FOR_VARIANT' " + $PRINTF "%s: \'%s\' " "$variant" "$JVM_FEATURES_FOR_VARIANT" done $ECHO "" diff --git a/make/autoconf/jdk-options.m4 b/make/autoconf/jdk-options.m4 index 79e44dd4ad1..289ed935fdf 100644 --- a/make/autoconf/jdk-options.m4 +++ b/make/autoconf/jdk-options.m4 @@ -405,10 +405,19 @@ AC_DEFUN_ONCE([JDKOPT_SETUP_CODE_COVERAGE], JCOV_FILTERS="$with_jcov_filters" fi fi + + UTIL_ARG_WITH(NAME: jcov-modules, TYPE: string, + DEFAULT: [], RESULT: JCOV_MODULES_COMMMA_SEPARATED, + DESC: [which modules to include in jcov (comma-separated)], + OPTIONAL: true) + + # Replace "," with " ". + JCOV_MODULES=${JCOV_MODULES_COMMMA_SEPARATED//,/ } AC_SUBST(JCOV_ENABLED) AC_SUBST(JCOV_HOME) AC_SUBST(JCOV_INPUT_JDK) AC_SUBST(JCOV_FILTERS) + AC_SUBST(JCOV_MODULES) ]) ################################################################################ @@ -520,8 +529,21 @@ AC_DEFUN_ONCE([JDKOPT_SETUP_UNDEFINED_BEHAVIOR_SANITIZER], # Silence them for now. UBSAN_CHECKS="-fsanitize=undefined -fsanitize=float-divide-by-zero -fno-sanitize=shift-base -fno-sanitize=alignment \ $ADDITIONAL_UBSAN_CHECKS" - UBSAN_CFLAGS="$UBSAN_CHECKS -Wno-stringop-truncation -Wno-format-overflow -Wno-array-bounds -Wno-stringop-overflow -fno-omit-frame-pointer -DUNDEFINED_BEHAVIOR_SANITIZER" + UBSAN_CFLAGS="$UBSAN_CHECKS -Wno-array-bounds -fno-omit-frame-pointer -DUNDEFINED_BEHAVIOR_SANITIZER" + if test "x$TOOLCHAIN_TYPE" = "xgcc"; then + UBSAN_CFLAGS="$UBSAN_CFLAGS -Wno-format-overflow -Wno-stringop-overflow -Wno-stringop-truncation" + fi UBSAN_LDFLAGS="$UBSAN_CHECKS" + # On AIX, the llvm_symbolizer is not found out of the box, so we have to provide the + # full qualified llvm_symbolizer path in the __ubsan_default_options() function in + # make/data/ubsan/ubsan_default_options.c. To get it there we compile our sources + # with an additional define LLVM_SYMBOLIZER, which we set here. + # To calculate the correct llvm_symbolizer path we can use the location of the compiler, because + # their relation is fixed. + if test "x$TOOLCHAIN_TYPE" = "xclang" && test "x$OPENJDK_TARGET_OS" = "xaix"; then + UBSAN_CFLAGS="$UBSAN_CFLAGS -fno-sanitize=function,vptr -DLLVM_SYMBOLIZER=$(dirname $(dirname $CC))/tools/ibm-llvm-symbolizer" + UBSAN_LDFLAGS="$UBSAN_LDFLAGS -fno-sanitize=function,vptr -Wl,-bbigtoc" + fi UTIL_ARG_ENABLE(NAME: ubsan, DEFAULT: false, RESULT: UBSAN_ENABLED, DESC: [enable UndefinedBehaviorSanitizer], CHECK_AVAILABLE: [ diff --git a/make/autoconf/lib-tests.m4 b/make/autoconf/lib-tests.m4 index d2a4fcbb191..9eb5ee5a046 100644 --- a/make/autoconf/lib-tests.m4 +++ b/make/autoconf/lib-tests.m4 @@ -28,7 +28,7 @@ ################################################################################ # Minimum supported versions -JTREG_MINIMUM_VERSION=7.5.1 +JTREG_MINIMUM_VERSION=7.5.2 GTEST_MINIMUM_VERSION=1.14.0 ################################################################################ diff --git a/make/autoconf/spec.gmk.template b/make/autoconf/spec.gmk.template index 907a60290ec..e720916d88a 100644 --- a/make/autoconf/spec.gmk.template +++ b/make/autoconf/spec.gmk.template @@ -1,5 +1,5 @@ # -# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2011, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -454,6 +454,7 @@ JCOV_ENABLED := @JCOV_ENABLED@ JCOV_HOME := @JCOV_HOME@ JCOV_INPUT_JDK := @JCOV_INPUT_JDK@ JCOV_FILTERS := @JCOV_FILTERS@ +JCOV_MODULES := @JCOV_MODULES@ # AddressSanitizer ASAN_ENABLED := @ASAN_ENABLED@ @@ -846,10 +847,12 @@ SVE_CFLAGS := @SVE_CFLAGS@ JDK_IMAGE_SUBDIR := jdk JRE_IMAGE_SUBDIR := jre JCOV_IMAGE_SUBDIR := jdk-jcov +STATIC_JDK_IMAGE_SUBDIR := static-jdk # Colon left out to be able to override output dir for bootcycle-images JDK_IMAGE_DIR = $(IMAGES_OUTPUTDIR)/$(JDK_IMAGE_SUBDIR) JRE_IMAGE_DIR = $(IMAGES_OUTPUTDIR)/$(JRE_IMAGE_SUBDIR) +STATIC_JDK_IMAGE_DIR = $(IMAGES_OUTPUTDIR)/$(STATIC_JDK_IMAGE_SUBDIR) JCOV_IMAGE_DIR = $(IMAGES_OUTPUTDIR)/$(JCOV_IMAGE_SUBDIR) # Test image, as above @@ -929,6 +932,7 @@ DOCS_JAVASE_BUNDLE_NAME := javase-$(BASE_NAME)_doc-api-spec$(DEBUG_PART).tar.gz DOCS_REFERENCE_BUNDLE_NAME := jdk-reference-$(BASE_NAME)_doc-api-spec$(DEBUG_PART).tar.gz STATIC_LIBS_BUNDLE_NAME := jdk-$(BASE_NAME)_bin-static-libs$(DEBUG_PART).tar.gz STATIC_LIBS_GRAAL_BUNDLE_NAME := jdk-$(BASE_NAME)_bin-static-libs-graal$(DEBUG_PART).tar.gz +STATIC_JDK_BUNDLE_NAME := static-jdk-$(BASE_NAME)_bin$(DEBUG_PART).$(JDK_BUNDLE_EXTENSION) JCOV_BUNDLE_NAME := jdk-jcov-$(BASE_NAME)_bin$(DEBUG_PART).$(JDK_BUNDLE_EXTENSION) JDK_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(JDK_BUNDLE_NAME) @@ -939,6 +943,7 @@ TEST_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(TEST_BUNDLE_NAME) DOCS_JDK_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(DOCS_JDK_BUNDLE_NAME) DOCS_JAVASE_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(DOCS_JAVASE_BUNDLE_NAME) DOCS_REFERENCE_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(DOCS_REFERENCE_BUNDLE_NAME) +STATIC_JDK_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(STATIC_JDK_BUNDLE_NAME) JCOV_BUNDLE := $(BUNDLES_OUTPUTDIR)/$(JCOV_BUNDLE_NAME) # This macro is called to allow inclusion of closed source counterparts. diff --git a/make/autoconf/util_paths.m4 b/make/autoconf/util_paths.m4 index 9e3e5472c9e..40864680aad 100644 --- a/make/autoconf/util_paths.m4 +++ b/make/autoconf/util_paths.m4 @@ -1,5 +1,5 @@ # -# Copyright (c) 2011, 2024, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2011, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -58,21 +58,32 @@ AC_DEFUN([UTIL_PREPEND_TO_PATH], # 2) The path will be absolute, and it will be in unix-style (on # cygwin). # $1: The name of the variable to fix -# $2: if NOFAIL, errors will be silently ignored +# $2: if NOFAIL, if the path cannot be resolved then errors will not be +# reported and an empty path will be set AC_DEFUN([UTIL_FIXUP_PATH], [ # Only process if variable expands to non-empty path="[$]$1" if test "x$path" != x; then if test "x$OPENJDK_BUILD_OS" = "xwindows"; then - if test "x$2" = "xNOFAIL"; then - quiet_option="-q" + imported_path=`$FIXPATH_BASE -q import "$path"` + if test $? -ne 0 || test ! -e $imported_path; then + if test "x$2" != "xNOFAIL"; then + AC_MSG_NOTICE([The path of $1, which is given as "$path", can not be properly resolved.]) + AC_MSG_NOTICE([Please see the section "Special Considerations" in building.md.]) + AC_MSG_NOTICE([This is the error message given by fixpath:]) + # Rerun fixpath without -q to get an error message + $FIXPATH_BASE import "$path" + AC_MSG_ERROR([Cannot continue]) + else + imported_path="" + fi fi - imported_path=`$FIXPATH_BASE $quiet_option import "$path"` - $FIXPATH_BASE verify "$imported_path" + + $FIXPATH_BASE -q verify "$imported_path" if test $? -ne 0; then if test "x$2" != "xNOFAIL"; then - AC_MSG_ERROR([The path of $1, which resolves as "$path", could not be imported.]) + AC_MSG_ERROR([The path of $1, which resolves as "$path", could not be verified.]) else imported_path="" fi @@ -83,7 +94,7 @@ AC_DEFUN([UTIL_FIXUP_PATH], if test "x$imported_path_lower" != "x$orig_path_lower"; then $1="$imported_path" fi - else + else # non-Windows [ if [[ "$path" =~ " " ]]; then ] if test "x$2" != "xNOFAIL"; then AC_MSG_NOTICE([The path of $1, which resolves as "$path", is invalid.]) @@ -186,7 +197,6 @@ AC_DEFUN([UTIL_CHECK_WINENV_EXEC_TYPE], # it need to be in the PATH. # $1: The name of the variable to fix # $2: Where to look for the command (replaces $PATH) -# $3: set to NOFIXPATH to skip prefixing FIXPATH, even if needed on platform AC_DEFUN([UTIL_FIXUP_EXECUTABLE], [ input="[$]$1" @@ -233,15 +243,19 @@ AC_DEFUN([UTIL_FIXUP_EXECUTABLE], # This is a path with slashes, don't look at $PATH if test "x$OPENJDK_BUILD_OS" = "xwindows"; then # fixpath.sh import will do all heavy lifting for us - new_path=`$FIXPATH_BASE import "$path"` + new_path=`$FIXPATH_BASE -q import "$path"` - if test ! -e $new_path; then + if test $? -ne 0 || test ! -e $new_path; then # It failed, but maybe spaces were part of the path and not separating # the command and argument. Retry using that assumption. - new_path=`$FIXPATH_BASE import "$input"` - if test ! -e $new_path; then - AC_MSG_NOTICE([The command for $1, which resolves as "$input", can not be found.]) - AC_MSG_ERROR([Cannot locate $input]) + new_path=`$FIXPATH_BASE -q import "$input"` + if test $? -ne 0 || test ! -e $new_path; then + AC_MSG_NOTICE([The command for $1, which is given as "$input", can not be properly resolved.]) + AC_MSG_NOTICE([Please see the section "Special Considerations" in building.md.]) + AC_MSG_NOTICE([This is the error message given by fixpath:]) + # Rerun fixpath without -q to get an error message + $FIXPATH_BASE import "$input" + AC_MSG_ERROR([Cannot continue]) fi # It worked, clear all "arguments" arguments="" @@ -282,10 +296,6 @@ AC_DEFUN([UTIL_FIXUP_EXECUTABLE], fi fi - if test "x$3" = xNOFIXPATH; then - fixpath_prefix="" - fi - # Now join together the path and the arguments once again new_complete="$fixpath_prefix$new_path$arguments" $1="$new_complete" @@ -353,7 +363,15 @@ AC_DEFUN([UTIL_SETUP_TOOL], else # Otherwise we believe it is a complete path. Use it as it is. if test ! -x "$tool_command" && test ! -x "${tool_command}.exe"; then - AC_MSG_ERROR([User supplied tool $1="$tool_command" does not exist or is not executable]) + # Maybe the path had spaces in it; try again with the entire argument + if test ! -x "$tool_override" && test ! -x "${tool_override}.exe"; then + AC_MSG_ERROR([User supplied tool $1="$tool_override" does not exist or is not executable]) + else + # We successfully located the executable assuming the spaces were part of the path. + # We can't combine using paths with spaces and arguments, so assume tool_args is empty. + tool_command="$tool_override" + tool_args="" + fi fi if test ! -x "$tool_command"; then tool_command="${tool_command}.exe" @@ -379,7 +397,6 @@ AC_DEFUN([UTIL_SETUP_TOOL], # $1: variable to set # $2: executable name (or list of names) to look for # $3: [path] -# $4: set to NOFIXPATH to skip prefixing FIXPATH, even if needed on platform AC_DEFUN([UTIL_LOOKUP_PROGS], [ UTIL_SETUP_TOOL($1, [ @@ -421,10 +438,8 @@ AC_DEFUN([UTIL_LOOKUP_PROGS], # If we have FIXPATH enabled, strip all instances of it and prepend # a single one, to avoid double fixpath prefixing. - if test "x$4" != xNOFIXPATH; then - [ if [[ $FIXPATH != "" && $result =~ ^"$FIXPATH " ]]; then ] - result="\$FIXPATH ${result#"$FIXPATH "}" - fi + [ if [[ $FIXPATH != "" && $result =~ ^"$FIXPATH " ]]; then ] + result="\$FIXPATH ${result#"$FIXPATH "}" fi AC_MSG_RESULT([$result]) break 2; @@ -515,6 +530,24 @@ AC_DEFUN([UTIL_ADD_FIXPATH], fi ]) +################################################################################ +# Return a path to the executable binary from a command line, stripping away +# any FIXPATH prefix or arguments. The resulting value can be checked for +# existence using "test -e". The result is returned in a variable named +# "$1_EXECUTABLE". +# +# $1: variable describing the command to get the binary for +AC_DEFUN([UTIL_GET_EXECUTABLE], +[ + # Strip the FIXPATH prefix, if any + fixpath_stripped="[$]$1" + [ if [[ $FIXPATH != "" && $fixpath_stripped =~ ^"$FIXPATH " ]]; then ] + fixpath_stripped="${fixpath_stripped#"$FIXPATH "}" + fi + # Remove any arguments following the binary + $1_EXECUTABLE="${fixpath_stripped%% *}" +]) + ################################################################################ AC_DEFUN([UTIL_REMOVE_SYMBOLIC_LINKS], [ diff --git a/make/common/FindTests.gmk b/make/common/FindTests.gmk index 41cf08d9e48..517bb2973f4 100644 --- a/make/common/FindTests.gmk +++ b/make/common/FindTests.gmk @@ -59,14 +59,14 @@ ifeq ($(GENERATE_FIND_TESTS_FILE), true) $(call MakeTargetDir) ( $(foreach root, $(JTREG_TESTROOTS), \ $(ECHO) ""; \ - $(ECHO) -n "$(root)_JTREG_TEST_GROUPS := "; \ + $(PRINTF) "\n%s_JTREG_TEST_GROUPS := " "$(root)"; \ $(SED) -n -e 's/^\#.*//g' -e 's/\([^ ]*\)\w*=.*/\1/gp' \ $($(root)_JTREG_GROUP_FILES) \ | $(SORT) -u | $(TR) '\n' ' ' ; \ ) \ ) > $@ $(ECHO) "" >> $@ - $(ECHO) -n "MAKE_TEST_TARGETS := " >> $@ + $(PRINTF) "MAKE_TEST_TARGETS := " >> $@ $(MAKE) -s --no-print-directory $(MAKE_ARGS) \ SPEC=$(SPEC) -f $(TOPDIR)/test/make/TestMake.gmk print-targets \ TARGETS_FILE=$@ diff --git a/make/common/JavaCompilation.gmk b/make/common/JavaCompilation.gmk index 70b3557baea..c5a74413de1 100644 --- a/make/common/JavaCompilation.gmk +++ b/make/common/JavaCompilation.gmk @@ -80,15 +80,13 @@ endef # # The sed expression does this: # 1. Add a backslash before any :, = or ! that do not have a backslash already. -# 2. Apply the file unicode2x.sed which does a whole bunch of \u00XX to \xXX -# conversions. -# 3. Delete all lines starting with #. -# 4. Delete empty lines. -# 5. Append lines ending with \ with the next line. -# 6. Remove leading and trailing white space. Note that tabs must be explicit +# 2. Delete all lines starting with #. +# 3. Delete empty lines. +# 4. Append lines ending with \ with the next line. +# 5. Remove leading and trailing white space. Note that tabs must be explicit # as sed on macosx does not understand '\t'. -# 7. Replace the first \= with just =. -# 8. Finally it's all sorted to create a stable output. +# 6. Replace the first \= with just =. +# 7. Finally it's all sorted to create a stable output. # # It is assumed that = is the character used for separating names and values. define add_file_to_clean @@ -108,7 +106,6 @@ define add_file_to_clean ( $(CAT) $$< && $(ECHO) "" ) \ | $(SED) -e 's/\([^\\]\):/\1\\:/g' -e 's/\([^\\]\)=/\1\\=/g' \ -e 's/\([^\\]\)!/\1\\!/g' -e 's/^[ ]*#.*/#/g' \ - | $(SED) -f "$$(TOPDIR)/make/common/support/unicode2x.sed" \ | $(SED) -e '/^#/d' -e '/^$$$$/d' \ -e :a -e '/\\$$$$/N; s/\\\n//; ta' \ -e 's/^[ ]*//;s/[ ]*$$$$//' \ @@ -155,6 +152,7 @@ endef # INCLUDE_FILES "com/sun/SolarisFoobar.java" means only compile this file! # EXCLUDE_FILES "com/sun/SolarisFoobar.java" means do not compile this particular file! # "SolarisFoobar.java" means do not compile SolarisFoobar, wherever it is found. +# EXCLUDE_PATTERNS Exclude files matching any of these substrings # EXTRA_FILES List of extra source files to include in compilation. Can be used to # specify files that need to be generated by other rules first. # HEADERS path to directory where all generated c-headers are written. @@ -265,10 +263,12 @@ define SetupJavaCompilationBody endif # Tell javac to do exactly as told and no more - PARANOIA_FLAGS := -implicit:none -Xprefer:source -XDignore.symbol.file=true -encoding ascii + PARANOIA_FLAGS := -implicit:none -Xprefer:source -XDignore.symbol.file=true $1_FLAGS += -g -Xlint:all $$($1_TARGET_RELEASE) $$(PARANOIA_FLAGS) $1_FLAGS += $$($1_JAVAC_FLAGS) + # Set character encoding in source + $1_FLAGS += -encoding utf-8 ifeq ($$(JAVA_WARNINGS_AS_ERRORS), true) $1_FLAGS += -Werror @@ -333,6 +333,20 @@ define SetupJavaCompilationBody $1_INCLUDE_PATTERN += $$(foreach i, $$($1_SRC), $$(addprefix $$i/, $$(addsuffix /%, $$($1_INCLUDES)))) endif + ifneq ($$($1_EXCLUDE_PATTERNS), ) + # We must not match the exclude pattern against the src roots, so first + # strip the src prefixes from the absolute file paths in SRCS. + $1_SRCS_WITHOUT_ROOTS := $$(foreach i, $$($1_SRC), \ + $$(patsubst $$i/%,%, $$(filter $$i/%, $$($1_SRCS)))) + $1_EXCLUDE_PATTERNS_WITHOUT_ROOTS := $$(call containing, \ + $$($1_EXCLUDE_PATTERNS), $$($1_SRCS_WITHOUT_ROOTS)) + # The add back all possible src prefixes; this will generate more paths + # than really exists, but it does not matter since we will use this as + # input to filter-out. + $1_EXCLUDE_PATTERN += $$(foreach i, $$($1_SRC), $$(addprefix $$i/, \ + $$($1_EXCLUDE_PATTERNS_WITHOUT_ROOTS))) + endif + # Apply include/exclude patterns to java sources ifneq ($$($1_EXCLUDE_PATTERN), ) $1_SRCS := $$(filter-out $$($1_EXCLUDE_PATTERN), $$($1_SRCS)) diff --git a/make/common/JdkNativeCompilation.gmk b/make/common/JdkNativeCompilation.gmk index 372ad39305c..0285669ffd8 100644 --- a/make/common/JdkNativeCompilation.gmk +++ b/make/common/JdkNativeCompilation.gmk @@ -227,6 +227,8 @@ endef GLOBAL_VERSION_INFO_RESOURCE := $(TOPDIR)/src/java.base/windows/native/common/version.rc +# \xA9 is the copyright symbol in ANSI encoding (Windows-1252), which rc.exe +# assumes the resource file is in. JDK_RCFLAGS=$(RCFLAGS) \ -D"JDK_VERSION_STRING=$(VERSION_STRING)" \ -D"JDK_COMPANY=$(JDK_RC_COMPANY_NAME)" \ diff --git a/make/common/MakeFileStart.gmk b/make/common/MakeFileStart.gmk index f1dd0abb792..f18c623d3e8 100644 --- a/make/common/MakeFileStart.gmk +++ b/make/common/MakeFileStart.gmk @@ -47,7 +47,7 @@ endif # We need spec.gmk to get $(TOPDIR) include $(SPEC) -THIS_MAKEFILE := $(patsubst make/%,%,$(patsubst $(TOPDIR)/%,%,$(THIS_MAKEFILE_PATH))) +THIS_MAKEFILE := $(patsubst make/%,%,$(patsubst $(TOPDIR_ALT)/make/%,%,$(patsubst $(TOPDIR)/%,%,$(THIS_MAKEFILE_PATH)))) ifeq ($(LOG_FLOW), true) $(info :Enter $(THIS_MAKEFILE)) diff --git a/make/common/MakeIncludeStart.gmk b/make/common/MakeIncludeStart.gmk index d09f027c1d3..3904633f9f2 100644 --- a/make/common/MakeIncludeStart.gmk +++ b/make/common/MakeIncludeStart.gmk @@ -29,7 +29,7 @@ # Get the next to last word (by prepending a padding element) THIS_INCLUDE_PATH := $(word $(words ${MAKEFILE_LIST}),padding ${MAKEFILE_LIST}) -THIS_INCLUDE := $(patsubst $(TOPDIR)/make/%,%,$(THIS_INCLUDE_PATH)) +THIS_INCLUDE := $(patsubst $(TOPDIR_ALT)/make/%,%,$(patsubst $(TOPDIR)/make/%,%,$(THIS_INCLUDE_PATH))) # Print an indented message, also counting the top-level makefile as a level ifneq ($(INCLUDE_GUARD_$(THIS_INCLUDE)), true) diff --git a/make/common/Modules.gmk b/make/common/Modules.gmk index 663e9075cf8..725424d7618 100644 --- a/make/common/Modules.gmk +++ b/make/common/Modules.gmk @@ -180,7 +180,7 @@ ifeq ($(GENERATE_MODULE_DEPS_FILE), true) $(call MakeTargetDir) $(RM) $@ $(foreach m, $(MODULE_INFOS), \ - ( $(ECHO) -n "DEPS_$(call GetModuleNameFromModuleInfo, $m) := " && \ + ( $(PRINTF) "DEPS_%s := " "$(call GetModuleNameFromModuleInfo, $m)" && \ $(AWK) -v MODULE=$(call GetModuleNameFromModuleInfo, $m) ' \ BEGIN { if (MODULE != "java.base") printf(" java.base"); } \ /^ *requires/ { sub(/;/, ""); \ @@ -194,7 +194,7 @@ ifeq ($(GENERATE_MODULE_DEPS_FILE), true) gsub(/\r/, ""); \ printf(" %s", $$0) } \ END { printf("\n") }' $m && \ - $(ECHO) -n "TRANSITIVE_MODULES_$(call GetModuleNameFromModuleInfo, $m) := " && \ + $(PRINTF) "TRANSITIVE_MODULES_%s := " "$(call GetModuleNameFromModuleInfo, $m)" && \ $(AWK) -v MODULE=$(call GetModuleNameFromModuleInfo, $m) ' \ BEGIN { if (MODULE != "java.base") printf(" java.base"); } \ /^ *requires *transitive/ { \ diff --git a/make/common/native/Paths.gmk b/make/common/native/Paths.gmk index ee097b2e134..bdb8828eb32 100644 --- a/make/common/native/Paths.gmk +++ b/make/common/native/Paths.gmk @@ -128,10 +128,9 @@ define SetupSourceFiles # Extract the C/C++ files. ifneq ($$($1_EXCLUDE_PATTERNS), ) # We must not match the exclude pattern against the src root(s). - $1_SRCS_WITHOUT_ROOTS := $$($1_SRCS) - $$(foreach i, $$($1_SRC), $$(eval $1_SRCS_WITHOUT_ROOTS := $$(patsubst \ - $$i/%,%, $$($1_SRCS_WITHOUT_ROOTS)))) - $1_ALL_EXCLUDE_FILES := $$(call containing, $$($1_EXCLUDE_PATTERNS), \ + $1_SRCS_WITHOUT_ROOTS := $$(foreach i, $$($1_SRC), \ + $$(patsubst $$i/%,%, $$(filter $$i/%, $$($1_SRCS)))) + $1_ALL_EXCLUDE_FILES := $$(call containing, $$($1_EXCLUDE_PATTERNS), \ $$($1_SRCS_WITHOUT_ROOTS)) endif ifneq ($$($1_EXCLUDE_FILES), ) diff --git a/make/common/support/unicode2x.sed b/make/common/support/unicode2x.sed deleted file mode 100644 index 5188b97fe03..00000000000 --- a/make/common/support/unicode2x.sed +++ /dev/null @@ -1,100 +0,0 @@ -s/\\u0020/\x20/g -s/\\u003A/\x3A/g -s/\\u006B/\x6B/g -s/\\u0075/\x75/g -s/\\u00A0/\xA0/g -s/\\u00A3/\xA3/g -s/\\u00B0/\xB0/g -s/\\u00B7/\xB7/g -s/\\u00BA/\xBA/g -s/\\u00BF/\xBF/g -s/\\u00C0/\xC0/g -s/\\u00C1/\xC1/g -s/\\u00C2/\xC2/g -s/\\u00C4/\xC4/g -s/\\u00C5/\xC5/g -s/\\u00C8/\xC8/g -s/\\u00C9/\xC9/g -s/\\u00CA/\xCA/g -s/\\u00CD/\xCD/g -s/\\u00CE/\xCE/g -s/\\u00D3/\xD3/g -s/\\u00D4/\xD4/g -s/\\u00D6/\xD6/g -s/\\u00DA/\xDA/g -s/\\u00DC/\xDC/g -s/\\u00DD/\xDD/g -s/\\u00DF/\xDF/g -s/\\u00E0/\xE0/g -s/\\u00E1/\xE1/g -s/\\u00E2/\xE2/g -s/\\u00E3/\xE3/g -s/\\u00E4/\xE4/g -s/\\u00E5/\xE5/g -s/\\u00E6/\xE6/g -s/\\u00E7/\xE7/g -s/\\u00E8/\xE8/g -s/\\u00E9/\xE9/g -s/\\u00EA/\xEA/g -s/\\u00EB/\xEB/g -s/\\u00EC/\xEC/g -s/\\u00ED/\xED/g -s/\\u00EE/\xEE/g -s/\\u00EF/\xEF/g -s/\\u00F1/\xF1/g -s/\\u00F2/\xF2/g -s/\\u00F3/\xF3/g -s/\\u00F4/\xF4/g -s/\\u00F5/\xF5/g -s/\\u00F6/\xF6/g -s/\\u00F9/\xF9/g -s/\\u00FA/\xFA/g -s/\\u00FC/\xFC/g -s/\\u0020/\x20/g -s/\\u003f/\x3f/g -s/\\u006f/\x6f/g -s/\\u0075/\x75/g -s/\\u00a0/\xa0/g -s/\\u00a3/\xa3/g -s/\\u00b0/\xb0/g -s/\\u00ba/\xba/g -s/\\u00bf/\xbf/g -s/\\u00c1/\xc1/g -s/\\u00c4/\xc4/g -s/\\u00c5/\xc5/g -s/\\u00c8/\xc8/g -s/\\u00c9/\xc9/g -s/\\u00ca/\xca/g -s/\\u00cd/\xcd/g -s/\\u00d6/\xd6/g -s/\\u00dc/\xdc/g -s/\\u00dd/\xdd/g -s/\\u00df/\xdf/g -s/\\u00e0/\xe0/g -s/\\u00e1/\xe1/g -s/\\u00e2/\xe2/g -s/\\u00e3/\xe3/g -s/\\u00e4/\xe4/g -s/\\u00e5/\xe5/g -s/\\u00e7/\xe7/g -s/\\u00e8/\xe8/g -s/\\u00e9/\xe9/g -s/\\u00ea/\xea/g -s/\\u00eb/\xeb/g -s/\\u00ec/\xec/g -s/\\u00ed/\xed/g -s/\\u00ee/\xee/g -s/\\u00ef/\xef/g -s/\\u00f0/\xf0/g -s/\\u00f1/\xf1/g -s/\\u00f2/\xf2/g -s/\\u00f3/\xf3/g -s/\\u00f4/\xf4/g -s/\\u00f5/\xf5/g -s/\\u00f6/\xf6/g -s/\\u00f7/\xf7/g -s/\\u00f8/\xf8/g -s/\\u00f9/\xf9/g -s/\\u00fa/\xfa/g -s/\\u00fc/\xfc/g -s/\\u00ff/\xff/g diff --git a/make/conf/github-actions.conf b/make/conf/github-actions.conf index 27845ffbd7a..d2b6cd23128 100644 --- a/make/conf/github-actions.conf +++ b/make/conf/github-actions.conf @@ -26,7 +26,7 @@ # Versions and download locations for dependencies used by GitHub Actions (GHA) GTEST_VERSION=1.14.0 -JTREG_VERSION=7.5.1+1 +JTREG_VERSION=7.5.2+1 LINUX_X64_BOOT_JDK_EXT=tar.gz LINUX_X64_BOOT_JDK_URL=https://download.java.net/java/GA/jdk24/1f9ff9062db4449d8ca828c504ffae90/36/GPL/openjdk-24_linux-x64_bin.tar.gz diff --git a/make/conf/jib-profiles.js b/make/conf/jib-profiles.js index aa4d846280e..91876878046 100644 --- a/make/conf/jib-profiles.js +++ b/make/conf/jib-profiles.js @@ -241,10 +241,10 @@ var getJibProfilesCommon = function (input, data) { // List of the main profile names used for iteration common.main_profile_names = [ - "linux-x64", "linux-x86", "macosx-x64", "macosx-aarch64", + "macosx-x64", "macosx-aarch64", "windows-x64", "windows-aarch64", - "linux-aarch64", "linux-arm32", "linux-ppc64le", "linux-s390x", - "linux-riscv64" + "linux-x64", "linux-aarch64", + "linux-arm32", "linux-ppc64le", "linux-s390x", "linux-riscv64" ]; // These are the base settings for all the main build profiles. @@ -254,7 +254,6 @@ var getJibProfilesCommon = function (input, data) { configure_args: concat( "--with-exclude-translations=es,fr,it,ko,pt_BR,sv,ca,tr,cs,sk,ja_JP_A,ja_JP_HA,ja_JP_HI,ja_JP_I,zh_TW,zh_HK", "--disable-jvm-feature-shenandoahgc", - "--disable-cds-archive-coh", versionArgs(input, common)) }; @@ -283,9 +282,6 @@ var getJibProfilesCommon = function (input, data) { labels: "open" }; - common.configure_args_64bit = ["--with-target-bits=64"]; - common.configure_args_32bit = ["--with-target-bits=32"]; - /** * Define common artifacts template for all main profiles * @param o - Object containing data for artifacts @@ -412,58 +408,34 @@ var getJibProfilesProfiles = function (input, common, data) { // Main SE profiles var profiles = { - - "linux-x64": { - target_os: "linux", - target_cpu: "x64", - dependencies: ["devkit", "gtest", "build_devkit", "graphviz", "pandoc", "tidy"], - configure_args: concat( - (input.build_cpu == "x64" ? common.configure_args_64bit - : "--openjdk-target=x86_64-linux-gnu"), - "--with-zlib=system", "--disable-dtrace", - (isWsl(input) ? [ "--host=x86_64-unknown-linux-gnu", - "--build=x86_64-unknown-linux-gnu" ] : [])), - }, - - "linux-x86": { - target_os: "linux", - target_cpu: "x86", - build_cpu: "x64", - dependencies: ["devkit", "gtest", "libffi"], - configure_args: concat(common.configure_args_32bit, [ - "--with-jvm-variants=minimal,server", - "--with-zlib=system", - "--with-libffi=" + input.get("libffi", "home_path"), - "--enable-libffi-bundling", - "--enable-fallback-linker" - ]) - }, - "macosx-x64": { target_os: "macosx", target_cpu: "x64", dependencies: ["devkit", "gtest", "graphviz", "pandoc", "tidy"], - configure_args: concat(common.configure_args_64bit, "--with-zlib=system", + configure_args: [ + "--with-zlib=system", "--with-macosx-version-max=11.00.00", "--enable-compatible-cds-alignment", // Use system SetFile instead of the one in the devkit as the // devkit one may not work on Catalina. - "SETFILE=/usr/bin/SetFile"), + "SETFILE=/usr/bin/SetFile" + ], }, "macosx-aarch64": { target_os: "macosx", target_cpu: "aarch64", dependencies: ["devkit", "gtest", "graphviz", "pandoc", "tidy"], - configure_args: concat(common.configure_args_64bit, - "--with-macosx-version-max=11.00.00"), + configure_args: [ + "--with-macosx-version-max=11.00.00" + ], }, "windows-x64": { target_os: "windows", target_cpu: "x64", dependencies: ["devkit", "gtest", "pandoc"], - configure_args: concat(common.configure_args_64bit), + configure_args: [], }, "windows-aarch64": { @@ -475,7 +447,19 @@ var getJibProfilesProfiles = function (input, common, data) { ], }, - "linux-aarch64": { + "linux-x64": { + target_os: "linux", + target_cpu: "x64", + dependencies: ["devkit", "gtest", "build_devkit", "graphviz", "pandoc", "tidy"], + configure_args: concat( + "--with-zlib=system", + "--disable-dtrace", + (cross_compiling ? [ "--openjdk-target=x86_64-linux-gnu" ] : []), + (isWsl(input) ? [ "--host=x86_64-unknown-linux-gnu", + "--build=x86_64-unknown-linux-gnu" ] : [])), + }, + + "linux-aarch64": { target_os: "linux", target_cpu: "aarch64", dependencies: ["devkit", "gtest", "build_devkit", "graphviz", "pandoc", "tidy"], @@ -492,8 +476,10 @@ var getJibProfilesProfiles = function (input, common, data) { build_cpu: "x64", dependencies: ["devkit", "gtest", "build_devkit"], configure_args: [ - "--openjdk-target=arm-linux-gnueabihf", "--with-freetype=bundled", - "--with-abi-profile=arm-vfp-hflt", "--disable-warnings-as-errors" + "--openjdk-target=arm-linux-gnueabihf", + "--with-freetype=bundled", + "--with-abi-profile=arm-vfp-hflt", + "--disable-warnings-as-errors" ], }, @@ -503,7 +489,8 @@ var getJibProfilesProfiles = function (input, common, data) { build_cpu: "x64", dependencies: ["devkit", "gtest", "build_devkit"], configure_args: [ - "--openjdk-target=ppc64le-linux-gnu", "--with-freetype=bundled", + "--openjdk-target=ppc64le-linux-gnu", + "--with-freetype=bundled", "--disable-warnings-as-errors" ], }, @@ -514,7 +501,8 @@ var getJibProfilesProfiles = function (input, common, data) { build_cpu: "x64", dependencies: ["devkit", "gtest", "build_devkit"], configure_args: [ - "--openjdk-target=s390x-linux-gnu", "--with-freetype=bundled", + "--openjdk-target=s390x-linux-gnu", + "--with-freetype=bundled", "--disable-warnings-as-errors" ], }, @@ -525,7 +513,8 @@ var getJibProfilesProfiles = function (input, common, data) { build_cpu: "x64", dependencies: ["devkit", "gtest", "build_devkit"], configure_args: [ - "--openjdk-target=riscv64-linux-gnu", "--with-freetype=bundled", + "--openjdk-target=riscv64-linux-gnu", + "--with-freetype=bundled", "--disable-warnings-as-errors" ], }, @@ -586,24 +575,24 @@ var getJibProfilesProfiles = function (input, common, data) { target_os: "linux", target_cpu: "x64", dependencies: ["devkit", "gtest", "libffi"], - configure_args: concat(common.configure_args_64bit, [ + configure_args: [ "--with-zlib=system", "--with-jvm-variants=zero", "--with-libffi=" + input.get("libffi", "home_path"), "--enable-libffi-bundling", - ]) + ] }, "linux-aarch64-zero": { target_os: "linux", target_cpu: "aarch64", dependencies: ["devkit", "gtest", "libffi"], - configure_args: concat(common.configure_args_64bit, [ + configure_args: [ "--with-zlib=system", "--with-jvm-variants=zero", "--with-libffi=" + input.get("libffi", "home_path"), "--enable-libffi-bundling" - ]) + ] }, "linux-x86-zero": { @@ -611,12 +600,13 @@ var getJibProfilesProfiles = function (input, common, data) { target_cpu: "x86", build_cpu: "x64", dependencies: ["devkit", "gtest", "libffi"], - configure_args: concat(common.configure_args_32bit, [ + configure_args: [ + "--with-target-bits=32", "--with-zlib=system", "--with-jvm-variants=zero", "--with-libffi=" + input.get("libffi", "home_path"), "--enable-libffi-bundling" - ]) + ] } } profiles = concatObjects(profiles, zeroProfiles); @@ -635,8 +625,10 @@ var getJibProfilesProfiles = function (input, common, data) { target_os: "linux", target_cpu: "x64", dependencies: ["devkit", "gtest"], - configure_args: concat(common.configure_args_64bit, - "--with-zlib=system", "--disable-precompiled-headers"), + configure_args: [ + "--with-zlib=system", + "--disable-precompiled-headers" + ], }, }; profiles = concatObjects(profiles, noPchProfiles); @@ -693,9 +685,6 @@ var getJibProfilesProfiles = function (input, common, data) { "linux-x64": { platform: "linux-x64", }, - "linux-x86": { - platform: "linux-x86", - }, "macosx-x64": { platform: "macos-x64", jdk_subdir: "jdk-" + data.version + ".jdk/Contents/Home", @@ -872,7 +861,8 @@ var getJibProfilesProfiles = function (input, common, data) { profiles[cmpBaselineName].configure_args = concat( profiles[cmpBaselineName].configure_args, "--with-hotspot-build-time=n/a", - "--disable-precompiled-headers"); + "--disable-precompiled-headers", + "--with-source-date=version"); // Do not inherit artifact definitions from base profile delete profiles[cmpBaselineName].artifacts; }); @@ -1089,8 +1079,8 @@ var getJibProfilesDependencies = function (input, common) { var devkit_platform_revisions = { linux_x64: "gcc14.2.0-OL6.4+1.0", - macosx: "Xcode14.3.1+1.0", - windows_x64: "VS2022-17.6.5+1.0", + macosx: "Xcode15.4+1.0", + windows_x64: "VS2022-17.13.2+1.0", linux_aarch64: "gcc14.2.0-OL7.6+1.0", linux_arm: "gcc8.2.0-Fedora27+1.0", linux_ppc64le: "gcc14.2.0-Fedora_41+1.0", @@ -1161,10 +1151,7 @@ var getJibProfilesDependencies = function (input, common) { organization: common.organization, ext: "tar.gz", module: "devkit-" + devkit_cross_prefix + devkit_platform, - revision: devkit_platform_revisions[devkit_platform], - environment: { - "DEVKIT_HOME": input.get("devkit", "home_path"), - } + revision: devkit_platform_revisions[devkit_platform] }, build_devkit: { @@ -1187,9 +1174,9 @@ var getJibProfilesDependencies = function (input, common) { jtreg: { server: "jpg", product: "jtreg", - version: "7.5.1", + version: "7.5.2", build_number: "1", - file: "bundles/jtreg-7.5.1+1.zip", + file: "bundles/jtreg-7.5.2+1.zip", environment_name: "JT_HOME", environment_path: input.get("jtreg", "home_path") + "/bin", configure_args: "--with-jtreg=" + input.get("jtreg", "home_path"), diff --git a/make/conf/version-numbers.conf b/make/conf/version-numbers.conf index ce9e32315a9..38d6e42dff9 100644 --- a/make/conf/version-numbers.conf +++ b/make/conf/version-numbers.conf @@ -26,17 +26,17 @@ # Default version, product, and vendor information to use, # unless overridden by configure -DEFAULT_VERSION_FEATURE=25 +DEFAULT_VERSION_FEATURE=26 DEFAULT_VERSION_INTERIM=0 DEFAULT_VERSION_UPDATE=0 DEFAULT_VERSION_PATCH=0 DEFAULT_VERSION_EXTRA1=0 DEFAULT_VERSION_EXTRA2=0 DEFAULT_VERSION_EXTRA3=0 -DEFAULT_VERSION_DATE=2025-09-16 -DEFAULT_VERSION_CLASSFILE_MAJOR=69 # "`$EXPR $DEFAULT_VERSION_FEATURE + 44`" +DEFAULT_VERSION_DATE=2026-03-17 +DEFAULT_VERSION_CLASSFILE_MAJOR=70 # "`$EXPR $DEFAULT_VERSION_FEATURE + 44`" DEFAULT_VERSION_CLASSFILE_MINOR=0 DEFAULT_VERSION_DOCS_API_SINCE=11 -DEFAULT_ACCEPTABLE_BOOT_VERSIONS="24 25" -DEFAULT_JDK_SOURCE_TARGET_VERSION=25 +DEFAULT_ACCEPTABLE_BOOT_VERSIONS="24 25 26" +DEFAULT_JDK_SOURCE_TARGET_VERSION=26 DEFAULT_PROMOTED_VERSION_PRE=ea diff --git a/make/data/ubsan/ubsan_default_options.c b/make/data/ubsan/ubsan_default_options.c index 011d1a675a9..05e4722e45a 100644 --- a/make/data/ubsan/ubsan_default_options.c +++ b/make/data/ubsan/ubsan_default_options.c @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022, 2023, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2022, 2025, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -43,6 +43,18 @@ #define ATTRIBUTE_USED #endif +// On AIX, the llvm_symbolizer is not found out of the box, so we have to provide the +// full qualified llvm_symbolizer path in the __ubsan_default_options() function. +// To get it here we compile our sources with an additional define LLVM_SYMBOLIZER +// containing the path, which we set in make/autoconf/jdk-options.m4. +#ifdef LLVM_SYMBOLIZER +#define _LLVM_SYMBOLIZER(X) ",external_symbolizer_path=" X_LLVM_SYMBOLIZER(X) +#define X_LLVM_SYMBOLIZER(X) #X +#else +#define LLVM_SYMBOLIZER +#define _LLVM_SYMBOLIZER(X) +#endif + // Override weak symbol exposed by UBSan to override default options. This is called by UBSan // extremely early during library loading, before main is called. We need to override the default // options because by default UBSan only prints a warning for each occurrence. We want jtreg tests @@ -50,5 +62,5 @@ // thread so it is easier to track down. You can override these options by setting the environment // variable UBSAN_OPTIONS. ATTRIBUTE_DEFAULT_VISIBILITY ATTRIBUTE_USED const char* __ubsan_default_options() { - return "halt_on_error=1,print_stacktrace=1"; + return "halt_on_error=1,print_stacktrace=1" _LLVM_SYMBOLIZER(LLVM_SYMBOLIZER); } diff --git a/make/devkit/Tools.gmk b/make/devkit/Tools.gmk index f4323f58638..1b9240df49c 100644 --- a/make/devkit/Tools.gmk +++ b/make/devkit/Tools.gmk @@ -39,6 +39,8 @@ # Fix this... # +uppercase = $(shell echo $1 | tr a-z A-Z) + $(info TARGET=$(TARGET)) $(info HOST=$(HOST)) $(info BUILD=$(BUILD)) @@ -91,99 +93,28 @@ endif ################################################################################ # Define external dependencies -# Latest that could be made to work. -GCC_VER := 14.2.0 -ifeq ($(GCC_VER), 14.2.0) - gcc_ver := gcc-14.2.0 - binutils_ver := binutils-2.43 - ccache_ver := ccache-4.10.2 - CCACHE_CMAKE_BASED := 1 - mpfr_ver := mpfr-4.2.1 - gmp_ver := gmp-6.3.0 - mpc_ver := mpc-1.3.1 - gdb_ver := gdb-15.2 - REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 -else ifeq ($(GCC_VER), 13.2.0) - gcc_ver := gcc-13.2.0 - binutils_ver := binutils-2.41 - ccache_ver := ccache-3.7.12 - mpfr_ver := mpfr-4.2.0 - gmp_ver := gmp-6.3.0 - mpc_ver := mpc-1.3.1 - gdb_ver := gdb-13.2 - REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 -else ifeq ($(GCC_VER), 11.3.0) - gcc_ver := gcc-11.3.0 - binutils_ver := binutils-2.39 - ccache_ver := ccache-3.7.12 - mpfr_ver := mpfr-4.1.1 - gmp_ver := gmp-6.2.1 - mpc_ver := mpc-1.2.1 - gdb_ver := gdb-11.2 - REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 -else ifeq ($(GCC_VER), 11.2.0) - gcc_ver := gcc-11.2.0 - binutils_ver := binutils-2.37 - ccache_ver := ccache-3.7.12 - mpfr_ver := mpfr-4.1.0 - gmp_ver := gmp-6.2.1 - mpc_ver := mpc-1.2.1 - gdb_ver := gdb-11.1 - REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 -else ifeq ($(GCC_VER), 10.3.0) - gcc_ver := gcc-10.3.0 - binutils_ver := binutils-2.36.1 - ccache_ver := ccache-3.7.11 - mpfr_ver := mpfr-4.1.0 - gmp_ver := gmp-6.2.0 - mpc_ver := mpc-1.1.0 - gdb_ver := gdb-10.1 - REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 -else ifeq ($(GCC_VER), 10.2.0) - gcc_ver := gcc-10.2.0 - binutils_ver := binutils-2.35 - ccache_ver := ccache-3.7.11 - mpfr_ver := mpfr-4.1.0 - gmp_ver := gmp-6.2.0 - mpc_ver := mpc-1.1.0 - gdb_ver := gdb-9.2 - REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 -else ifeq ($(GCC_VER), 9.2.0) - gcc_ver := gcc-9.2.0 - binutils_ver := binutils-2.34 - ccache_ver := ccache-3.7.3 - mpfr_ver := mpfr-3.1.5 - gmp_ver := gmp-6.1.2 - mpc_ver := mpc-1.0.3 - gdb_ver := gdb-8.3 -else ifeq ($(GCC_VER), 8.3.0) - gcc_ver := gcc-8.3.0 - binutils_ver := binutils-2.32 - ccache_ver := ccache-3.7.3 - mpfr_ver := mpfr-3.1.5 - gmp_ver := gmp-6.1.2 - mpc_ver := mpc-1.0.3 - gdb_ver := gdb-8.3 -else ifeq ($(GCC_VER), 7.3.0) - gcc_ver := gcc-7.3.0 - binutils_ver := binutils-2.30 - ccache_ver := ccache-3.3.6 - mpfr_ver := mpfr-3.1.5 - gmp_ver := gmp-6.1.2 - mpc_ver := mpc-1.0.3 - gdb_ver := gdb-8.1 -else ifeq ($(GCC_VER), 4.9.2) - gcc_ver := gcc-4.9.2 - binutils_ver := binutils-2.25 - ccache_ver := ccache-3.2.1 - mpfr_ver := mpfr-3.0.1 - gmp_ver := gmp-4.3.2 - mpc_ver := mpc-1.0.1 - gdb_ver := gdb-7.12.1 -else - $(error Unsupported GCC version) -endif +gcc_ver_only := 14.2.0 +binutils_ver_only := 2.43 +ccache_ver_only := 4.10.2 +CCACHE_CMAKE_BASED := 1 +mpfr_ver_only := 4.2.1 +gmp_ver_only := 6.3.0 +mpc_ver_only := 1.3.1 +gdb_ver_only := 15.2 + +dependencies := gcc binutils ccache mpfr gmp mpc gdb +$(foreach dep,$(dependencies),$(eval $(dep)_ver := $(dep)-$($(dep)_ver_only))) + +GCC := http://ftp.gnu.org/pub/gnu/gcc/$(gcc_ver)/$(gcc_ver).tar.xz +BINUTILS := http://ftp.gnu.org/pub/gnu/binutils/$(binutils_ver).tar.gz +CCACHE := https://github.com/ccache/ccache/releases/download/v$(ccache_ver_only)/$(ccache_ver).tar.xz +MPFR := https://www.mpfr.org/$(mpfr_ver)/$(mpfr_ver).tar.bz2 +GMP := http://ftp.gnu.org/pub/gnu/gmp/$(gmp_ver).tar.bz2 +MPC := http://ftp.gnu.org/pub/gnu/mpc/$(mpc_ver).tar.gz +GDB := http://ftp.gnu.org/gnu/gdb/$(gdb_ver).tar.xz + +REQUIRED_MIN_MAKE_MAJOR_VERSION := 4 ifneq ($(REQUIRED_MIN_MAKE_MAJOR_VERSION),) MAKE_MAJOR_VERSION := $(word 1,$(subst ., ,$(MAKE_VERSION))) SUPPORTED_MAKE_VERSION := $(shell [ $(MAKE_MAJOR_VERSION) -ge $(REQUIRED_MIN_MAKE_MAJOR_VERSION) ] && echo true) @@ -192,17 +123,6 @@ ifneq ($(REQUIRED_MIN_MAKE_MAJOR_VERSION),) endif endif -ccache_ver_only := $(patsubst ccache-%,%,$(ccache_ver)) - - -GCC := http://ftp.gnu.org/pub/gnu/gcc/$(gcc_ver)/$(gcc_ver).tar.xz -BINUTILS := http://ftp.gnu.org/pub/gnu/binutils/$(binutils_ver).tar.gz -CCACHE := https://github.com/ccache/ccache/releases/download/v$(ccache_ver_only)/$(ccache_ver).tar.xz -MPFR := https://www.mpfr.org/${mpfr_ver}/${mpfr_ver}.tar.bz2 -GMP := http://ftp.gnu.org/pub/gnu/gmp/${gmp_ver}.tar.bz2 -MPC := http://ftp.gnu.org/pub/gnu/mpc/${mpc_ver}.tar.gz -GDB := http://ftp.gnu.org/gnu/gdb/${gdb_ver}.tar.xz - # RPMs used by all BASE_OS RPM_LIST := \ $(KERNEL_HEADERS_RPM) \ @@ -297,7 +217,7 @@ define Download endef # Download and unpack all source packages -$(foreach p,GCC BINUTILS CCACHE MPFR GMP MPC GDB,$(eval $(call Download,$(p)))) +$(foreach dep,$(dependencies),$(eval $(call Download,$(call uppercase,$(dep))))) ################################################################################ # Unpack RPMS @@ -374,7 +294,7 @@ endif ################################################################################ # Define marker files for each source package to be compiled -$(foreach t,binutils mpfr gmp mpc gcc ccache gdb,$(eval $(t) = $(TARGETDIR)/$($(t)_ver).done)) +$(foreach dep,$(dependencies),$(eval $(dep) = $(TARGETDIR)/$($(dep)_ver).done)) ################################################################################ @@ -721,12 +641,12 @@ ifeq ($(TARGET), $(HOST)) ln -s $(TARGET)-$* $@ missing-links := $(addprefix $(PREFIX)/bin/, \ - addr2line ar as c++ c++filt dwp elfedit g++ gcc gcc-$(GCC_VER) gprof ld ld.bfd \ + addr2line ar as c++ c++filt dwp elfedit g++ gcc gcc-$(gcc_ver_only) gprof ld ld.bfd \ ld.gold nm objcopy objdump ranlib readelf size strings strip) endif # Add link to work around "plugin needed to handle lto object" (JDK-8344272) -$(PREFIX)/lib/bfd-plugins/liblto_plugin.so: $(PREFIX)/libexec/gcc/$(TARGET)/$(GCC_VER)/liblto_plugin.so +$(PREFIX)/lib/bfd-plugins/liblto_plugin.so: $(PREFIX)/libexec/gcc/$(TARGET)/$(gcc_ver_only)/liblto_plugin.so @echo 'Creating missing $(@F) soft link' @mkdir -p $(@D) ln -s $$(realpath -s --relative-to=$(@D) $<) $@ diff --git a/make/devkit/createAutoconfBundle.sh b/make/devkit/createAutoconfBundle.sh index 7363b9cd8a7..ebe9c427f76 100644 --- a/make/devkit/createAutoconfBundle.sh +++ b/make/devkit/createAutoconfBundle.sh @@ -1,6 +1,6 @@ #!/bin/bash -e # -# Copyright (c) 2018, 2024, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2018, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -24,10 +24,21 @@ # questions. # -# Create a bundle in the current directory, containing what's needed to run +# Create a bundle in OpenJDK build folder, containing what's needed to run # the 'autoconf' program by the OpenJDK build. To override TARGET_PLATFORM # just set the variable before running this script. +# This script fetches sources from network so make sure your proxy is setup appropriately. + +# colored print to highlight some of the logs +function print_log() +{ + Color_Cyan='\033[1;36m' # Cyan + Color_Off='\033[0m' # Reset color + printf "${Color_Cyan}> $1${Color_Off}\n" +} + + # Autoconf depends on m4, so download and build that first. AUTOCONF_VERSION=2.69 M4_VERSION=1.4.18 @@ -58,11 +69,12 @@ MODULE_NAME=autoconf-$TARGET_PLATFORM-$AUTOCONF_VERSION+$PACKAGE_VERSION BUNDLE_NAME=$MODULE_NAME.tar.gz SCRIPT_DIR="$(cd "$(dirname $0)" > /dev/null && pwd)" -OUTPUT_ROOT="${SCRIPT_DIR}/../../build/autoconf" +BASEDIR="$(cd "$SCRIPT_DIR/../.." > /dev/null && pwd)" +OUTPUT_ROOT="$BASEDIR/build/autoconf" -cd $OUTPUT_ROOT IMAGE_DIR=$OUTPUT_ROOT/$MODULE_NAME mkdir -p $IMAGE_DIR/usr +cd $OUTPUT_ROOT # Download and build m4 @@ -76,7 +88,7 @@ elif test "x$TARGET_PLATFORM" = xcygwin_x86; then cp /usr/bin/m4 $IMAGE_DIR/usr/bin elif test "x$TARGET_PLATFORM" = xlinux_x64; then M4_VERSION=1.4.13-5 - wget http://yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/m4-$M4_VERSION.el6.x86_64.rpm + wget https://yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/getPackage/m4-$M4_VERSION.el6.x86_64.rpm cd $IMAGE_DIR rpm2cpio $OUTPUT_ROOT/m4-$M4_VERSION.el6.x86_64.rpm | cpio -d -i elif test "x$TARGET_PLATFORM" = xlinux_x86; then @@ -85,27 +97,38 @@ elif test "x$TARGET_PLATFORM" = xlinux_x86; then cd $IMAGE_DIR rpm2cpio $OUTPUT_ROOT/m4-$M4_VERSION.el6.i686.rpm | cpio -d -i else + print_log "m4: download" wget https://ftp.gnu.org/gnu/m4/m4-$M4_VERSION.tar.gz - tar xzf m4-$M4_VERSION.tar.gz + tar -xzf m4-$M4_VERSION.tar.gz cd m4-$M4_VERSION + print_log "m4: configure" ./configure --prefix=$IMAGE_DIR/usr CFLAGS="-w -Wno-everything" + print_log "m4: make" make + print_log "m4: make install" make install cd .. fi # Download and build autoconf +print_log "autoconf: download" wget https://ftp.gnu.org/gnu/autoconf/autoconf-$AUTOCONF_VERSION.tar.gz -tar xzf autoconf-$AUTOCONF_VERSION.tar.gz +tar -xzf autoconf-$AUTOCONF_VERSION.tar.gz cd autoconf-$AUTOCONF_VERSION +print_log "autoconf: configure" ./configure --prefix=$IMAGE_DIR/usr M4=$IMAGE_DIR/usr/bin/m4 +print_log "autoconf: make" make +print_log "autoconf: make install" make install cd .. +# The resulting scripts from installation folder use absolute paths to reference other files within installation folder +print_log "replace absolue paths from installation files with a relative ." perl -pi -e "s!$IMAGE_DIR/!./!" $IMAGE_DIR/usr/bin/auto* $IMAGE_DIR/usr/share/autoconf/autom4te.cfg +print_log "creating $IMAGE_DIR/autoconf wrapper script" cat > $IMAGE_DIR/autoconf << EOF #!/bin/bash # Get an absolute path to this script @@ -123,6 +146,9 @@ PREPEND_INCLUDE="--prepend-include \$this_script_dir/usr/share/autoconf" exec \$this_script_dir/usr/bin/autoconf \$PREPEND_INCLUDE "\$@" EOF + chmod +x $IMAGE_DIR/autoconf + +print_log "archiving $IMAGE_DIR directory as $OUTPUT_ROOT/$BUNDLE_NAME" cd $IMAGE_DIR tar -cvzf $OUTPUT_ROOT/$BUNDLE_NAME * diff --git a/make/devkit/createWindowsDevkit.sh b/make/devkit/createWindowsDevkit.sh index 0646cb68ef4..757fb157ad4 100644 --- a/make/devkit/createWindowsDevkit.sh +++ b/make/devkit/createWindowsDevkit.sh @@ -56,16 +56,22 @@ BUILD_DIR="${SCRIPT_DIR}/../../build/devkit" UNAME_SYSTEM=`uname -s` UNAME_RELEASE=`uname -r` +UNAME_OS=`uname -o` # Detect cygwin or WSL IS_CYGWIN=`echo $UNAME_SYSTEM | grep -i CYGWIN` IS_WSL=`echo $UNAME_RELEASE | grep Microsoft` +IS_MSYS=`echo $UNAME_OS | grep -i Msys` +MSYS2_ARG_CONV_EXCL="*" # make "cmd.exe /c" work for msys2 +CMD_EXE="cmd.exe /c" if test "x$IS_CYGWIN" != "x"; then BUILD_ENV="cygwin" +elif test "x$IS_MSYS" != "x"; then + BUILD_ENV="cygwin" elif test "x$IS_WSL" != "x"; then BUILD_ENV="wsl" else - echo "Unknown environment; only Cygwin and WSL are supported." + echo "Unknown environment; only Cygwin/MSYS2/WSL are supported." exit 1 fi @@ -76,7 +82,7 @@ elif test "x$BUILD_ENV" = "xwsl"; then fi # Work around the insanely named ProgramFiles(x86) env variable -PROGRAMFILES_X86="$($WINDOWS_PATH_TO_UNIX_PATH "$(cmd.exe /c set | sed -n 's/^ProgramFiles(x86)=//p' | tr -d '\r')")" +PROGRAMFILES_X86="$($WINDOWS_PATH_TO_UNIX_PATH "$(${CMD_EXE} set | sed -n 's/^ProgramFiles(x86)=//p' | tr -d '\r')")" PROGRAMFILES="$($WINDOWS_PATH_TO_UNIX_PATH "$PROGRAMFILES")" case $VS_VERSION in @@ -99,13 +105,15 @@ esac # Find Visual Studio installation dir -VSNNNCOMNTOOLS=`cmd.exe /c echo %VS${VS_VERSION_NUM_NODOT}COMNTOOLS% | tr -d '\r'` +VSNNNCOMNTOOLS=`${CMD_EXE} echo %VS${VS_VERSION_NUM_NODOT}COMNTOOLS% | tr -d '\r'` +VSNNNCOMNTOOLS="$($WINDOWS_PATH_TO_UNIX_PATH "$VSNNNCOMNTOOLS")" if [ -d "$VSNNNCOMNTOOLS" ]; then - VS_INSTALL_DIR="$($WINDOWS_PATH_TO_UNIX_PATH "$VSNNNCOMNTOOLS/../..")" + VS_INSTALL_DIR="$VSNNNCOMNTOOLS/../.." else VS_INSTALL_DIR="${MSVC_PROGRAMFILES_DIR}/Microsoft Visual Studio/$VS_VERSION" VS_INSTALL_DIR="$(ls -d "${VS_INSTALL_DIR}/"{Community,Professional,Enterprise} 2>/dev/null | head -n1)" fi +echo "VSNNNCOMNTOOLS: $VSNNNCOMNTOOLS" echo "VS_INSTALL_DIR: $VS_INSTALL_DIR" # Extract semantic version @@ -180,7 +188,11 @@ cp $DEVKIT_ROOT/VC/redist/arm64/$MSVCP_DLL $DEVKIT_ROOT/VC/bin/arm64 ################################################################################ # Copy SDK files -SDK_INSTALL_DIR="$PROGRAMFILES_X86/Windows Kits/$SDK_VERSION" +SDK_INSTALL_DIR=`${CMD_EXE} echo %WindowsSdkDir% | tr -d '\r'` +SDK_INSTALL_DIR="$($WINDOWS_PATH_TO_UNIX_PATH "$SDK_INSTALL_DIR")" +if [ ! -d "$SDK_INSTALL_DIR" ]; then + SDK_INSTALL_DIR="$PROGRAMFILES_X86/Windows Kits/$SDK_VERSION" +fi echo "SDK_INSTALL_DIR: $SDK_INSTALL_DIR" SDK_FULL_VERSION="$(ls "$SDK_INSTALL_DIR/bin" | sort -r -n | head -n1)" diff --git a/make/hotspot/lib/JvmFeatures.gmk b/make/hotspot/lib/JvmFeatures.gmk index 0a897230f83..0fd1c752174 100644 --- a/make/hotspot/lib/JvmFeatures.gmk +++ b/make/hotspot/lib/JvmFeatures.gmk @@ -125,9 +125,11 @@ endif ifneq ($(call check-jvm-feature, cds), true) JVM_CFLAGS_FEATURES += -DINCLUDE_CDS=0 JVM_EXCLUDE_FILES += \ + aotCodeCache.cpp \ classLoaderDataShared.cpp \ classLoaderExt.cpp \ - systemDictionaryShared.cpp + systemDictionaryShared.cpp \ + trainingData.cpp JVM_EXCLUDE_PATTERNS += cds/ endif diff --git a/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java b/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java index 1b930ca7527..fa1b33bb03e 100644 --- a/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java +++ b/make/jdk/src/classes/build/tools/classlist/HelloClasslist.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2016, 2025, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -59,6 +59,7 @@ public class HelloClasslist { private static final Logger LOGGER = Logger.getLogger("Hello"); + @SuppressWarnings("restricted") public static void main(String ... args) throws Throwable { FileSystems.getDefault(); @@ -141,6 +142,7 @@ public static void main(String ... args) throws Throwable { HelloClasslist.class.getMethod("staticMethod_V").invoke(null); var obj = HelloClasslist.class.getMethod("staticMethod_L_L", Object.class).invoke(null, instance); HelloClasslist.class.getField("field").get(instance); + MethodHandles.Lookup.ClassOption.class.getEnumConstants(); // A selection of trivial and relatively common MH operations invoke(MethodHandles.identity(double.class), 1.0); @@ -160,6 +162,9 @@ record B(int b) { } case B b -> b.b; default -> 17; }; + // record run-time methods + o.equals(new B(5)); + o.hashCode(); LOGGER.log(Level.FINE, "Value: " + value); // The Striped64$Cell is loaded rarely only when there's a contention among diff --git a/make/jdk/src/classes/build/tools/generatecharacter/GenerateCharacter.java b/make/jdk/src/classes/build/tools/generatecharacter/GenerateCharacter.java index 8d6a703f1c9..a2f37db72d9 100644 --- a/make/jdk/src/classes/build/tools/generatecharacter/GenerateCharacter.java +++ b/make/jdk/src/classes/build/tools/generatecharacter/GenerateCharacter.java @@ -1,5 +1,5 @@ /* - * Copyright (c) 2002, 2023, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2002, 2025, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -1190,7 +1190,7 @@ static void genTable(StringBuffer result, String name, if (Csyntax) result.append(" static "); else - result.append(" static final "); + result.append(" @Stable static final "); result.append(atype); result.append(" ").append(name).append("["); if (Csyntax) @@ -1347,7 +1347,7 @@ else if (bits == 64) { } static void genCaseMapTableDeclaration(StringBuffer result) { - result.append(" static final char[][][] charMap;\n"); + result.append(" @Stable static final char[][][] charMap;\n"); } static void genCaseMapTable(StringBuffer result, SpecialCaseMap[] specialCaseMaps){ diff --git a/make/jdk/src/classes/build/tools/pandocfilter/PandocFilter.java b/make/jdk/src/classes/build/tools/pandocfilter/PandocFilter.java index 64eeaaa36df..ebb49613e53 100644 --- a/make/jdk/src/classes/build/tools/pandocfilter/PandocFilter.java +++ b/make/jdk/src/classes/build/tools/pandocfilter/PandocFilter.java @@ -1,3 +1,26 @@ +/* + * Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved. + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + */ + package build.tools.pandocfilter; import build.tools.pandocfilter.json.JSON; diff --git a/make/jdk/src/classes/build/tools/taglet/PreviewNote.java b/make/jdk/src/classes/build/tools/taglet/PreviewNote.java deleted file mode 100644 index ee3f9bea527..00000000000 --- a/make/jdk/src/classes/build/tools/taglet/PreviewNote.java +++ /dev/null @@ -1,127 +0,0 @@ -/* - * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved. - * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. - * - * This code is free software; you can redistribute it and/or modify it - * under the terms of the GNU General Public License version 2 only, as - * published by the Free Software Foundation. Oracle designates this - * particular file as subject to the "Classpath" exception as provided - * by Oracle in the LICENSE file that accompanied this code. - * - * This code is distributed in the hope that it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License - * version 2 for more details (a copy is included in the LICENSE file that - * accompanied this code). - * - * You should have received a copy of the GNU General Public License version - * 2 along with this work; if not, write to the Free Software Foundation, - * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. - * - * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA - * or visit www.oracle.com if you need additional information or have any - * questions. - */ - -package build.tools.taglet; - -import java.util.EnumSet; -import java.util.List; -import java.util.Set; - - -import javax.lang.model.element.Element; -import javax.tools.Diagnostic; - - -import com.sun.source.doctree.DocTree; -import com.sun.source.doctree.UnknownInlineTagTree; -import jdk.javadoc.doclet.Doclet; -import jdk.javadoc.doclet.DocletEnvironment; -import jdk.javadoc.doclet.Reporter; -import jdk.javadoc.doclet.StandardDoclet; -import jdk.javadoc.doclet.Taglet; - -import static com.sun.source.doctree.DocTree.Kind.UNKNOWN_INLINE_TAG; - -/** - * An inline tag to insert a note formatted as preview note. - * The tag can be used as follows: - * - *- * {@previewNote jep-number [Preview note heading]} - * Preview note content - * {@previewNote} - *- * - */ -public class PreviewNote implements Taglet { - - static final String TAG_NAME = "previewNote"; - Reporter reporter = null; - - @Override - public void init(DocletEnvironment env, Doclet doclet) { - if (doclet instanceof StandardDoclet stdoclet) { - reporter = stdoclet.getReporter(); - } - } - - /** - * Returns the set of locations in which the tag may be used. - */ - @Override - public SetgetAllowedLocations() { - return EnumSet.allOf(Taglet.Location.class); - } - - @Override - public boolean isInlineTag() { - return true; - } - - @Override - public String getName() { - return TAG_NAME; - } - - @Override - public String toString(List tags, Element elem) { - - for (DocTree tag : tags) { - if (tag.getKind() == UNKNOWN_INLINE_TAG) { - UnknownInlineTagTree inlineTag = (UnknownInlineTagTree) tag; - String[] content = inlineTag.getContent().toString().trim().split("\\s+", 2); - if (!content[0].isBlank()) { - StringBuilder sb = new StringBuilder(""" - - """); - if (content.length == 2) { - sb.append(""" -- """; - } - } - } - - if (reporter == null) { - throw new IllegalArgumentException("@" + TAG_NAME + " taglet content must be begin or end"); - } - reporter.print(Diagnostic.Kind.ERROR, "@" + TAG_NAME + " taglet content must be begin or end"); - return ""; - } -} diff --git a/make/langtools/src/classes/build/tools/symbolgenerator/CreateSymbols.java b/make/langtools/src/classes/build/tools/symbolgenerator/CreateSymbols.java index db8924c79fb..fe5938ce0e3 100644 --- a/make/langtools/src/classes/build/tools/symbolgenerator/CreateSymbols.java +++ b/make/langtools/src/classes/build/tools/symbolgenerator/CreateSymbols.java @@ -46,6 +46,8 @@ import java.io.StringWriter; import java.io.Writer; import java.lang.classfile.*; +import java.lang.classfile.TypeAnnotation.TargetInfo; +import java.lang.classfile.TypeAnnotation.TypePathComponent; import java.lang.classfile.attribute.*; import java.lang.classfile.constantpool.ClassEntry; import java.lang.classfile.constantpool.ConstantPoolBuilder; @@ -64,6 +66,9 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.attribute.BasicFileAttributes; +import java.time.Instant; +import java.time.LocalDateTime; +import java.time.ZoneOffset; import java.util.stream.Stream; import java.util.ArrayList; import java.util.Arrays; @@ -101,10 +106,7 @@ import com.sun.source.util.JavacTask; import com.sun.tools.javac.api.JavacTool; -import com.sun.tools.javac.jvm.Target; -import com.sun.tools.javac.util.Assert; import com.sun.tools.javac.util.Context; -import com.sun.tools.javac.util.Pair; import java.nio.file.DirectoryStream; import java.util.Optional; import java.util.function.Consumer; @@ -304,13 +306,18 @@ public void createSymbols(String ctDescriptionFileExtra, String ctDescriptionFil "Ljdk/internal/ValueBased;"; private static final String VALUE_BASED_ANNOTATION_INTERNAL = "Ljdk/internal/ValueBased+Annotation;"; + private static final String REQUIRES_IDENTITY_ANNOTATION = + "Ljdk/internal/RequiresIdentity;"; + private static final String REQUIRES_IDENTITY_ANNOTATION_INTERNAL = + "Ljdk/internal/RequiresIdentity+Annotation;"; public static final Set- """) - .append(content[1]) - .append(""" -- """); - } - sb.append(""" -- """); - return sb.toString(); - } else { - return """ --HARDCODED_ANNOTATIONS = new HashSet<>( List.of("Ljdk/Profile+Annotation;", "Lsun/Proprietary+Annotation;", PREVIEW_FEATURE_ANNOTATION_OLD, PREVIEW_FEATURE_ANNOTATION_NEW, VALUE_BASED_ANNOTATION, - RESTRICTED_ANNOTATION)); + RESTRICTED_ANNOTATION, + REQUIRES_IDENTITY_ANNOTATION)); private void stripNonExistentAnnotations(LoadDescriptions data) { Set allClasses = data.classes.name2Class.keySet(); @@ -339,10 +346,10 @@ private void stripNonExistentAnnotations(Set allClasses, List createAnnotations(List desc) { @@ -1021,6 +1034,12 @@ private Annotation createAnnotation(AnnotationDescription desc) { annotationType = VALUE_BASED_ANNOTATION_INTERNAL; } + if (REQUIRES_IDENTITY_ANNOTATION.equals(annotationType)) { + //the non-public RequiresIdentity annotation will not be available in ct.sym, + //replace with purely synthetic javac-internal annotation: + annotationType = REQUIRES_IDENTITY_ANNOTATION_INTERNAL; + } + if (RESTRICTED_ANNOTATION.equals(annotationType)) { //the non-public Restricted annotation will not be available in ct.sym, //replace with purely synthetic javac-internal annotation: @@ -1055,6 +1074,44 @@ private AnnotationValue createAttributeValue(Object value) { default -> throw new IllegalArgumentException(value.getClass().getName()); }; } + + private List createTypeAnnotations(List desc) { + return desc.stream().map(this::createTypeAnnotation).collect(Collectors.toList()); + } + + private TypeAnnotation createTypeAnnotation(TypeAnnotationDescription desc) { + Annotation baseAnn = createAnnotation(desc.annotation); + TargetInfo targetInfo = switch ((String) desc.targetInfo.get("targetType")) { + case "CLASS_TYPE_PARAMETER" -> //TODO: test! + TargetInfo.ofClassTypeParameter((int) desc.targetInfo.get("typeParameterIndex")); + case "METHOD_TYPE_PARAMETER" -> + TargetInfo.ofMethodTypeParameter((int) desc.targetInfo.get("typeParameterIndex")); + case "CLASS_EXTENDS" -> + TargetInfo.ofClassExtends((int) desc.targetInfo.get("supertypeIndex")); + case "CLASS_TYPE_PARAMETER_BOUND" -> + TargetInfo.ofClassTypeParameterBound((int) desc.targetInfo.get("typeParameterIndex"), + (int) desc.targetInfo.get("boundIndex")); + case "METHOD_TYPE_PARAMETER_BOUND" -> + TargetInfo.ofMethodTypeParameterBound((int) desc.targetInfo.get("typeParameterIndex"), + (int) desc.targetInfo.get("boundIndex")); + case "METHOD_RETURN" -> + TargetInfo.ofMethodReturn(); + case "METHOD_RECEIVER" -> + TargetInfo.ofMethodReceiver(); + case "METHOD_FORMAL_PARAMETER" -> + TargetInfo.ofMethodFormalParameter((int) desc.targetInfo.get("formalParameterIndex")); + case "THROWS" -> + TargetInfo.ofThrows((int) desc.targetInfo.get("throwsTargetIndex")); + case "FIELD" -> + TargetInfo.ofField(); + case String targetType -> + throw new IllegalStateException("Unsupported targetType: " + targetType); + }; + + List typePath = desc.typePath.stream().map(d -> TypePathComponent.of(TypePathComponent.Kind.valueOf(d.tag()), d.index())).toList(); + + return TypeAnnotation.of(targetInfo, typePath, baseAnn); + } // // @@ -2202,6 +2259,10 @@ private boolean readAttribute(FeatureDescription feature, Attribute attr) { chd.permittedSubclasses = a.permittedSubclasses().stream().map(ClassEntry::asInternalName).collect(Collectors.toList()); } case ModuleMainClassAttribute a -> ((ModuleHeaderDescription) feature).moduleMainClass = a.mainClass().asInternalName(); + case RuntimeInvisibleTypeAnnotationsAttribute a -> + feature.classTypeAnnotations = typeAnnotations2Descriptions(a.annotations()); + case RuntimeVisibleTypeAnnotationsAttribute a -> + feature.runtimeTypeAnnotations = typeAnnotations2Descriptions(a.annotations()); default -> throw new IllegalArgumentException("Unhandled attribute: " + attr.attributeName()); // Do nothing } @@ -2258,6 +2319,31 @@ private AnnotationDescription annotation2Description(java.lang.classfile.Annotat return new AnnotationDescription(annotationType, values); } + + private List typeAnnotations2Descriptions(List annos) { + return annos.stream().map(ta -> { + TypeAnnotationDescription desc = new TypeAnnotationDescription(); + desc.annotation = annotation2Description(ta.annotation()); + desc.targetInfo = new HashMap<>(); + desc.targetInfo.put("targetType", ta.targetInfo().targetType().name()); + switch (ta.targetInfo()) { + case TypeAnnotation.TypeParameterTarget tpt -> desc.targetInfo.put("typeParameterIndex", tpt.typeParameterIndex()); + case TypeAnnotation.SupertypeTarget st -> desc.targetInfo.put("supertypeIndex", st.supertypeIndex()); + case TypeAnnotation.TypeParameterBoundTarget tpbt -> { + desc.targetInfo.put("typeParameterIndex", tpbt.typeParameterIndex()); + desc.targetInfo.put("boundIndex", tpbt.boundIndex()); + } + case TypeAnnotation.EmptyTarget _ -> { + // nothing to write + } + case TypeAnnotation.FormalParameterTarget fpt -> desc.targetInfo.put("formalParameterIndex", fpt.formalParameterIndex()); + case TypeAnnotation.ThrowsTarget tt -> desc.targetInfo.put("throwsTargetIndex", tt.throwsTargetIndex()); + default -> throw new IllegalStateException(ta.targetInfo().targetType().name()); + } + desc.typePath = ta.targetPath().stream().map(tpc -> new TypeAnnotationDescription.TypePathComponentDesc(tpc.typePathKind().name(), tpc.typeArgumentIndex())).toList(); + return desc; + }).toList(); + } // protected boolean includeEffectiveAccess(ClassList classes, ClassDescription clazz) { @@ -2379,6 +2465,8 @@ static abstract class FeatureDescription { String versions = ""; List classAnnotations; List runtimeAnnotations; + List classTypeAnnotations; + List runtimeTypeAnnotations; protected void writeAttributes(Appendable output) throws IOException { if (flags != 0) @@ -2401,6 +2489,18 @@ protected void writeAttributes(Appendable output) throws IOException { output.append(quote(a.toString(), false)); } } + if (classTypeAnnotations != null && !classTypeAnnotations.isEmpty()) { + output.append(" classTypeAnnotations "); + for (TypeAnnotationDescription a : classTypeAnnotations) { + output.append(quote(a.toString(), false)); + } + } + if (runtimeTypeAnnotations != null && !runtimeTypeAnnotations.isEmpty()) { + output.append(" runtimeTypeAnnotations "); + for (TypeAnnotationDescription a : runtimeTypeAnnotations) { + output.append(quote(a.toString(), false)); + } + } } protected boolean shouldIgnore(String baselineVersion, String version) { @@ -2430,6 +2530,14 @@ protected void readAttributes(LineBasedReader reader) { if (inRuntimeAnnotations != null) { runtimeAnnotations = parseAnnotations(inRuntimeAnnotations, new int[1]); } + String inClassTypeAnnotations = reader.attributes.get("classTypeAnnotations"); + if (inClassTypeAnnotations != null) { + classTypeAnnotations = parseTypeAnnotations(inClassTypeAnnotations, new int[1]); + } + String inRuntimeTypeAnnotations = reader.attributes.get("runtimeTypeAnnotations"); + if (inRuntimeTypeAnnotations != null) { + runtimeTypeAnnotations = parseTypeAnnotations(inRuntimeTypeAnnotations, new int[1]); + } } public abstract boolean read(LineBasedReader reader) throws IOException; @@ -2442,6 +2550,8 @@ public int hashCode() { hash = 89 * hash + Objects.hashCode(this.signature); hash = 89 * hash + listHashCode(this.classAnnotations); hash = 89 * hash + listHashCode(this.runtimeAnnotations); + hash = 89 * hash + listHashCode(this.classTypeAnnotations); + hash = 89 * hash + listHashCode(this.runtimeTypeAnnotations); return hash; } @@ -2469,6 +2579,12 @@ public boolean equals(Object obj) { if (!listEquals(this.runtimeAnnotations, other.runtimeAnnotations)) { return false; } + if (!listEquals(this.classTypeAnnotations, other.classTypeAnnotations)) { + return false; + } + if (!listEquals(this.runtimeTypeAnnotations, other.runtimeTypeAnnotations)) { + return false; + } return true; } @@ -3273,6 +3389,8 @@ public int hashCode() { hash = 59 * hash + Objects.hashCode(this.descriptor); hash = 59 * hash + Objects.hashCode(this.thrownTypes); hash = 59 * hash + Objects.hashCode(this.annotationDefaultValue); + hash = 59 * hash + Objects.hashCode(this.classParameterAnnotations); + hash = 59 * hash + Objects.hashCode(this.runtimeParameterAnnotations); return hash; } @@ -3297,6 +3415,12 @@ public boolean equals(Object obj) { if (!Objects.equals(this.annotationDefaultValue, other.annotationDefaultValue)) { return false; } + if (!Objects.equals(this.classParameterAnnotations, other.classParameterAnnotations)) { + return false; + } + if (!Objects.equals(this.runtimeParameterAnnotations, other.runtimeParameterAnnotations)) { + return false; + } return true; } @@ -3624,6 +3748,40 @@ private static String dumpAnnotationValue(Object value) { } } + static final class TypeAnnotationDescription { + AnnotationDescription annotation; + Map typePath; + + public TypeAnnotationDescription() { + } + + public TypeAnnotationDescription(AnnotationDescription annotation, Map typePath) { + this.annotation = annotation; + this.targetInfo = targetInfo; + this.typePath = typePath; + } + + @Override + public String toString() { + return annotation.toString() + "{" + targetInfo.entrySet().stream().map(e -> e.getKey() + "=" + quote(printValue(e.getValue()), false)).collect(Collectors.joining(",")) + "}" + + (!typePath.isEmpty() ? "[" + typePath.stream().map(desc -> desc.tag + ":" + desc.index).collect(Collectors.joining(",")) + "]" : ""); + } + + private String printValue(Object obj) { + if (obj instanceof String s) { + return "\"" + s + "\""; + } else if (obj instanceof Integer i) { + return "I" + String.valueOf(i); + } else { + throw new IllegalStateException("Unsupported value: " + obj.getClass()); + } + } + + //TODO: path + record TypePathComponentDesc(String tag, int index) {} + } + static final class EnumConstant { String type; String constant; @@ -3963,23 +4121,69 @@ public static List parseAnnotations(String encoded, int[] private static AnnotationDescription parseAnnotation(String value, int[] valuePointer) { String className = className(value, valuePointer); - Map parseTypeAnnotations(String encoded, int[] pointer) { + List result = new ArrayList<>(); + + while (pointer[0] < encoded.length() && encoded.charAt(pointer[0]) == '@') { + pointer[0]++; + result.add(parseTypeAnnotation(encoded, pointer)); + } + + return result; + } + + private static TypeAnnotationDescription parseTypeAnnotation(String value, int[] valuePointer) { + AnnotationDescription ann = parseAnnotation(value, valuePointer); + Map typePath = new ArrayList<>(); + + if (valuePointer[0] < value.length() && value.charAt(valuePointer[0]) == '[') { + while (value.charAt(valuePointer[0]) != ']') { int nameStart = ++valuePointer[0]; - while (value.charAt(valuePointer[0]++) != '='); + while (value.charAt(valuePointer[0]++) != ':'); String name = value.substring(nameStart, valuePointer[0] - 1); - attribute2Value.put(name, parseAnnotationValue(value, valuePointer)); + typePath.add(new TypeAnnotationDescription.TypePathComponentDesc(name, Integer.parseInt(readDigits(value, valuePointer)))); } valuePointer[0]++; } - return new AnnotationDescription(className, attribute2Value); + return new TypeAnnotationDescription(ann, targetInfo, typePath); } // diff --git a/make/modules/java.base/Java.gmk b/make/modules/java.base/Java.gmk index 84344f93409..fc091377456 100644 --- a/make/modules/java.base/Java.gmk +++ b/make/modules/java.base/Java.gmk @@ -41,11 +41,6 @@ CLEAN += intrinsic.properties EXCLUDE_FILES += \ $(TOPDIR)/src/java.base/share/classes/jdk/internal/module/ModuleLoaderMap.java -EXCLUDES += java/lang/doc-files \ - java/lang/classfile/snippet-files \ - java/lang/classfile/components/snippet-files \ - java/lang/foreign/snippet-files - # Exclude BreakIterator classes that are just used in compile process to generate # data files and shouldn't go in the product EXCLUDE_FILES += sun/text/resources/BreakIteratorRules.java diff --git a/make/modules/java.base/Lib.gmk b/make/modules/java.base/Lib.gmk index 84ee309dadd..51d323a0344 100644 --- a/make/modules/java.base/Lib.gmk +++ b/make/modules/java.base/Lib.gmk @@ -158,6 +158,7 @@ endif $(eval $(call SetupJdkLibrary, BUILD_LIBSYSLOOKUP, \ NAME := syslookup, \ + EXTRA_HEADER_DIRS := java.base:libjava, \ LD_SET_ORIGIN := false, \ LDFLAGS_linux := -Wl$(COMMA)--no-as-needed, \ LDFLAGS_aix := -brtl -bexpfull, \ diff --git a/make/modules/java.compiler/Java.gmk b/make/modules/java.compiler/Java.gmk index cb720672639..d0a1fbf4cd5 100644 --- a/make/modules/java.compiler/Java.gmk +++ b/make/modules/java.compiler/Java.gmk @@ -32,6 +32,4 @@ DOCLINT += -Xdoclint:all/protected \ '-Xdoclint/package:java.*,javax.*' -EXCLUDES += javax/tools/snippet-files - ################################################################################ diff --git a/make/modules/java.desktop/Java.gmk b/make/modules/java.desktop/Java.gmk index 61c7fa44e0e..bab6186fb0d 100644 --- a/make/modules/java.desktop/Java.gmk +++ b/make/modules/java.desktop/Java.gmk @@ -31,15 +31,6 @@ DOCLINT += -Xdoclint:all/protected \ COPY += .gif .png .wav .txt .xml .css .pf CLEAN += iio-plugin.properties cursors.properties -EXCLUDES += \ - java/awt/doc-files \ - javax/swing/doc-files \ - javax/swing/text/doc-files \ - javax/swing/plaf/synth/doc-files \ - javax/swing/undo/doc-files \ - sun/awt/X11/doc-files \ - # - EXCLUDE_FILES += \ javax/swing/plaf/nimbus/InternalFrameTitlePanePainter.java \ javax/swing/plaf/nimbus/OptionPaneMessageAreaPainter.java \ diff --git a/make/modules/java.desktop/lib/AwtLibraries.gmk b/make/modules/java.desktop/lib/AwtLibraries.gmk index 5414c93c908..463e09e12dc 100644 --- a/make/modules/java.desktop/lib/AwtLibraries.gmk +++ b/make/modules/java.desktop/lib/AwtLibraries.gmk @@ -278,7 +278,6 @@ ifeq ($(call isTargetOs, windows macosx)+$(ENABLE_HEADLESS_ONLY), false+false) DISABLED_WARNINGS_gcc_X11TextRenderer_md.c := unused-but-set-variable, \ DISABLED_WARNINGS_gcc_XlibWrapper.c := type-limits pointer-to-int-cast, \ DISABLED_WARNINGS_gcc_XRBackendNative.c := maybe-uninitialized, \ - DISABLED_WARNINGS_gcc_XToolkit.c := unused-result, \ DISABLED_WARNINGS_gcc_XWindow.c := unused-function, \ DISABLED_WARNINGS_clang_awt_Taskbar.c := parentheses, \ DISABLED_WARNINGS_clang_gtk3_interface.c := unused-function parentheses, \ diff --git a/make/modules/java.desktop/lib/ClientLibraries.gmk b/make/modules/java.desktop/lib/ClientLibraries.gmk index a9511fba58b..dcb41defba3 100644 --- a/make/modules/java.desktop/lib/ClientLibraries.gmk +++ b/make/modules/java.desktop/lib/ClientLibraries.gmk @@ -87,7 +87,7 @@ $(eval $(call SetupJdkLibrary, BUILD_LIBLCMS, \ libawt/java2d \ java.base:libjvm, \ HEADERS_FROM_SRC := $(LIBLCMS_HEADERS_FROM_SRC), \ - DISABLED_WARNINGS_gcc := format-nonliteral stringop-truncation type-limits \ + DISABLED_WARNINGS_gcc := format-nonliteral stringop-truncation \ unused-variable, \ DISABLED_WARNINGS_clang := format-nonliteral, \ JDK_LIBS := libawt java.base:libjava, \ diff --git a/make/modules/jdk.jdi/Java.gmk b/make/modules/jdk.jdi/Java.gmk index d31008c318f..c5e3e715c1a 100644 --- a/make/modules/jdk.jdi/Java.gmk +++ b/make/modules/jdk.jdi/Java.gmk @@ -31,7 +31,6 @@ EXCLUDES += \ com/sun/tools/example/debug/bdi \ com/sun/tools/example/debug/event \ com/sun/tools/example/debug/gui \ - com/sun/jdi/doc-files \ # EXCLUDE_FILES += jdi-overview.html diff --git a/make/modules/jdk.jdwp.agent/Lib.gmk b/make/modules/jdk.jdwp.agent/Lib.gmk index 8fe7787d76e..6ae053d4bec 100644 --- a/make/modules/jdk.jdwp.agent/Lib.gmk +++ b/make/modules/jdk.jdwp.agent/Lib.gmk @@ -68,6 +68,7 @@ $(eval $(call SetupJdkLibrary, BUILD_LIBJDWP, \ DISABLED_WARNINGS_clang_threadControl.c := unused-but-set-variable \ unused-variable, \ DISABLED_WARNINGS_clang_utf_util.c := unused-but-set-variable, \ + DISABLED_WARNINGS_microsoft_debugInit.c := 5287, \ LDFLAGS := $(ICONV_LDFLAGS), \ EXTRA_HEADER_DIRS := \ include \ diff --git a/make/scripts/fixpath.sh b/make/scripts/fixpath.sh index 3a886fee07c..6a524df4c68 100644 --- a/make/scripts/fixpath.sh +++ b/make/scripts/fixpath.sh @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2020, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -157,11 +157,21 @@ function import_path() { if [[ $? -eq 0 && -e "$unixpath" ]]; then if [[ ! "$winpath" =~ ^"$ENVROOT"\\.*$ ]] ; then # If it is not in envroot, it's a generic windows path - if [[ ! $winpath =~ ^[-_.:\\a-zA-Z0-9]*$ ]] ; then + if [[ ! $winpath =~ ^[-_.:~+\\a-zA-Z0-9]*$ ]] ; then # Path has forbidden characters, rewrite as short name # This monster of a command uses the %~s support from cmd.exe to # reliably convert to short paths on all winenvs. shortpath="$($CMD /q /c for %I in \( "$winpath" \) do echo %~sI 2>/dev/null | tr -d \\n\\r)" + if [[ ! $shortpath =~ ^[-_.:~+\\a-zA-Z0-9]*$ ]] ; then + if [[ $QUIET != true ]]; then + echo fixpath: failure: Path "'"$path"'" could not be converted to short path >&2 + fi + if [[ $IGNOREFAILURES != true ]]; then + exit 1 + else + shortpath="" + fi + fi unixpath="$($PATHTOOL -u "$shortpath")" # unixpath is based on short name fi diff --git a/make/scripts/update_copyright_year.sh b/make/scripts/update_copyright_year.sh index bb61d48c91c..578ab4cbc99 100644 --- a/make/scripts/update_copyright_year.sh +++ b/make/scripts/update_copyright_year.sh @@ -1,7 +1,7 @@ #!/bin/bash -f # -# Copyright (c) 2010, 2024, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2010, 2025, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -41,10 +41,6 @@ set -e # To allow total changes counting shopt -s lastpipe -# Get an absolute path to this script, since that determines the top-level directory. -this_script_dir=`dirname $0` -this_script_dir=`cd $this_script_dir > /dev/null && pwd` - # Temp area tmp=/tmp/`basename $0`.${USER}.$$ rm -f -r ${tmp} @@ -98,10 +94,16 @@ while getopts "c:fhy:" option; do done # VCS check +git_installed=false +which git > /dev/null && git_installed=true +if [ "$git_installed" != "true" ]; then + echo "Error: This script requires git. Please install it." + exit 1 +fi git_found=false -[ -d "${this_script_dir}/../../.git" ] && git_found=true +git status &> /dev/null && git_found=true if [ "$git_found" != "true" ]; then - echo "Error: Please execute script from within make/scripts." + echo "Error: Please execute script from within a JDK git repository." exit 1 else echo "Using Git version control system" diff --git a/make/test/BuildMicrobenchmark.gmk b/make/test/BuildMicrobenchmark.gmk index 92f40472c3c..347ca44d25f 100644 --- a/make/test/BuildMicrobenchmark.gmk +++ b/make/test/BuildMicrobenchmark.gmk @@ -119,7 +119,6 @@ $(JMH_UNPACKED_JARS_DONE): $(JMH_RUNTIME_JARS) $(foreach jar, $(JMH_RUNTIME_JARS), \ $$($(UNZIP) -oq $(jar) -d $(JMH_UNPACKED_DIR))) $(RM) -r $(JMH_UNPACKED_DIR)/META-INF - $(RM) $(JMH_UNPACKED_DIR)/*.xml $(TOUCH) $@ # Copy dependency files for inclusion in the benchmark JARs diff --git a/src/demo/share/jfc/CodePointIM/README.html b/src/demo/share/jfc/CodePointIM/README.html index 4860e0f89c2..5d2173e9fdf 100644 --- a/src/demo/share/jfc/CodePointIM/README.html +++ b/src/demo/share/jfc/CodePointIM/README.html @@ -2,7 +2,7 @@ README - CodePointIM - +Code Point Input Method
diff --git a/src/demo/share/jfc/Metalworks/resources/HelpFiles/metal.html b/src/demo/share/jfc/Metalworks/resources/HelpFiles/metal.html index 6a42b1539ef..1347dda3fe2 100644 --- a/src/demo/share/jfc/Metalworks/resources/HelpFiles/metal.html +++ b/src/demo/share/jfc/Metalworks/resources/HelpFiles/metal.html @@ -1,6 +1,6 @@ - +Java L&F @@ -39,7 +39,7 @@The theme mechanism is designed to allow developers to create their own themes. For an example of this, see the themes which are included with Metalworks. Note, like all of the L&F packages, -the metal package is not yet frozen and the theme mechanism may change as +the metal package is not yet frozen and the theme mechanism may change as we get developer feedback on how to improve it.
@@ -53,7 +53,7 @@
JSlider slider = new JSlider();
@@ -64,7 +64,7 @@
-slider.putClientProperty("JSlider.isFilled", +slider.putClientProperty("JSlider.isFilled", Boolean.TRUE);
ToolBars
-Many popular applications support "roll-over" effects on buttons in toolbars. +
Many popular applications support "roll-over" effects on buttons in toolbars. The Java L&F provides an easy way to do this. Here is a code snippit:
@@ -72,7 +72,7 @@
@@ -98,7 +98,7 @@// add your buttons here
-toolbar.putClientProperty("JToolBar.isRollover", +toolbar.putClientProperty("JToolBar.isRollover", Boolean.TRUE);
// This will provide lines separating root notes // it is the default setting myTree.putClientProperty("JTree.lineStyle", - "Horizontal"); + "Horizontal");
- +
Back
diff --git a/src/demo/share/jfc/Metalworks/resources/HelpFiles/metalworks.html b/src/demo/share/jfc/Metalworks/resources/HelpFiles/metalworks.html
index df9a440b273..f8b9d0cc6e3 100644
--- a/src/demo/share/jfc/Metalworks/resources/HelpFiles/metalworks.html
+++ b/src/demo/share/jfc/Metalworks/resources/HelpFiles/metalworks.html
@@ -1,6 +1,6 @@
Metalworks diff --git a/src/demo/share/jfc/SwingSet2/resources/ant.html b/src/demo/share/jfc/SwingSet2/resources/ant.html index eb9619b6825..346678d2de4 100644 --- a/src/demo/share/jfc/SwingSet2/resources/ant.html +++ b/src/demo/share/jfc/SwingSet2/resources/ant.html @@ -1,7 +1,7 @@Untitled Document - + @@ -10,106 +10,106 @@
-+
-
This was a creature, more troublesom to be drawn, then any - of the rest, for I could not, for a good while, think of a way to make it +
This was a creature, more troublesom to be drawn, then any + of the rest, for I could not, for a good while, think of a way to make it suffer its body to ly quiet in a natural posture; but whil'st it was alive, - if its feet were fetter'd in Wax or Glew, it would so twist and wind its body, - that I could not any wayes get a good view of it; and if I killed it, its - body was so little, that I did often spoile the shape of it, before I could - throughly view it: for this is the nature of these minute Bodies, that as - soon, almost, as ever their life is destroy'd, their parts immediately shrivel, - and lose their beauty; and so is it also with small Plants, as I instanced + if its feet were fetter'd in Wax or Glew, it would so twist and wind its body, + that I could not any wayes get a good view of it; and if I killed it, its + body was so little, that I did often spoile the shape of it, before I could + throughly view it: for this is the nature of these minute Bodies, that as + soon, almost, as ever their life is destroy'd, their parts immediately shrivel, + and lose their beauty; and so is it also with small Plants, as I instanced before, in the description of Moss.
-And thence also is the reason of the variations in the beards - of wild Oats, and in those of Muskgrass seed, that their bodies, being exceeding - small, those small variations which are made in the surfaces of all bodies, - almost upon every change of Air, especially if the body be porous, do here - become sensible, where the whole body is so small, that it is almost nothing - but surface; for as in vegetable substances, I see no great reason to think, - that the moisture of the Aire (that, sticking to a wreath'd beard, does make - it untwist) should evaporate, or exhale away, any faster then the moisture - of other bodies, but rather that the avolation from, or access of moisture - to, the surfaces of bodies being much the same, those bodies become most - sensible of it, which have the least proportion of body to their surface. +
And thence also is the reason of the variations in the beards + of wild Oats, and in those of Muskgrass seed, that their bodies, being exceeding + small, those small variations which are made in the surfaces of all bodies, + almost upon every change of Air, especially if the body be porous, do here + become sensible, where the whole body is so small, that it is almost nothing + but surface; for as in vegetable substances, I see no great reason to think, + that the moisture of the Aire (that, sticking to a wreath'd beard, does make + it untwist) should evaporate, or exhale away, any faster then the moisture + of other bodies, but rather that the avolation from, or access of moisture + to, the surfaces of bodies being much the same, those bodies become most + sensible of it, which have the least proportion of body to their surface.
-So is it also with Animal substances; the dead body of an - Ant, or such little creature, does almost instantly shrivel and dry, and - your object shall be quite another thing, before you can half delineate - it, which proceeds not from the extraordinary exhalation, but from the small - proportion of body and juices, to the usual drying of bodies in the Air, +
So is it also with Animal substances; the dead body of an + Ant, or such little creature, does almost instantly shrivel and dry, and + your object shall be quite another thing, before you can half delineate + it, which proceeds not from the extraordinary exhalation, but from the small + proportion of body and juices, to the usual drying of bodies in the Air, especially if warm.
-For which inconvenience, where I could not otherwise remove - it, I thought of this expedient. I took the creature, I had design'd to delineate, - and put it into a drop of very well rectified spirit of Wine, this I found - would presently dispatch, as it were, the Animal, and being taken out of - it, and lay'd on a paper,the spirit of Wine would immediately fly away, - and leave the Animal dry, in its natural posture, or at least, in a constitution, - that it might easily with a pin be plac'd, in what posture you desired to - draw it, and the limbs would so remain, without either moving, or shriveling. +
For which inconvenience, where I could not otherwise remove + it, I thought of this expedient. I took the creature, I had design'd to delineate, + and put it into a drop of very well rectified spirit of Wine, this I found + would presently dispatch, as it were, the Animal, and being taken out of + it, and lay'd on a paper,the spirit of Wine would immediately fly away, + and leave the Animal dry, in its natural posture, or at least, in a constitution, + that it might easily with a pin be plac'd, in what posture you desired to + draw it, and the limbs would so remain, without either moving, or shriveling.
-And thus I dealt with this Ant, which I have here delineated, - which was one of many, of a very large kind, that inhabited under the Roots - of a Tree, from whence they would sally out in great parties, and make most - grievous havock of the Flowers and Fruits, in the ambient Garden, and return back +
And thus I dealt with this Ant, which I have here delineated, + which was one of many, of a very large kind, that inhabited under the Roots + of a Tree, from whence they would sally out in great parties, and make most + grievous havock of the Flowers and Fruits, in the ambient Garden, and return back again very expertly, by the same wayes and paths they went.
-It was more then half the bigness of an Earwig, of a dark - brown, or reddish colour, with long legs, on the hinder of which it would - stand up, and raise its head as high as it could above the ground, that it - might stare the further about it, just after the same manner as I have also - observ'd a hunting Spider to do: and putting my finger towards them, they - have at first all run towards it, till almost at it; and then they would stand - round about it, at a certain distance, and smell, as it were, and consider - whether they should any of them venture any further, till one more bold then - the rest venturing to climb it, all the rest, if I would have suffered them, - would have immediately followed : much such other seemingly rational actions - I have observ'd in this little Vermine with much pleasure, which would be - too long to be here related; those that desire more of them may satisfie +
It was more then half the bigness of an Earwig, of a dark + brown, or reddish colour, with long legs, on the hinder of which it would + stand up, and raise its head as high as it could above the ground, that it + might stare the further about it, just after the same manner as I have also + observ'd a hunting Spider to do: and putting my finger towards them, they + have at first all run towards it, till almost at it; and then they would stand + round about it, at a certain distance, and smell, as it were, and consider + whether they should any of them venture any further, till one more bold then + the rest venturing to climb it, all the rest, if I would have suffered them, + would have immediately followed : much such other seemingly rational actions + I have observ'd in this little Vermine with much pleasure, which would be + too long to be here related; those that desire more of them may satisfie their curiosity in Ligons History of the Barbadoes.
-Having insnar'd several of these into a small Box, I made - choice of the tallest grown among them, and separating it from the rest, - I gave it a Gill of Brandy, or Spirit of Wine, which after a while e'en knock'd - him down dead drunk, so that he became moveless, though at first putting - in he struggled for a pretty while very much, till at last, certain bubbles - issuing out of his mouth, it ceased to move; this (because I had before found - them quickly to recover again, if they were taken out presently) I suffered - to lye above an hour in the Spirit; and after I had taken it out, and put - its body and legs into a natural posture, remained moveless about an hour; - but then, upon a sudden, as if it had been awaken out of a drunken sleep, - it suddenly reviv'd and ran away; being caught, and serv'd as before, he - for a while continued struggling and striving, till at last there issued - several bubbles out of its mouth, and then, tanquam animam expirasset, he - remained moveless for a good while ; but at length again recovering, it was - again redipt, and suffered to lye some hours in the Spirit; notwithstanding - which, after it had layen dry some three or four hours, it again recovered - life and motion: Which kind of Experiments, if prosecuted, which they highly - deserve, seem to me of no inconsiderable use towards the invention of the - Latent Scheme, (as the Noble Ve rulam calls it) or the hidden, unknown Texture +
Having insnar'd several of these into a small Box, I made + choice of the tallest grown among them, and separating it from the rest, + I gave it a Gill of Brandy, or Spirit of Wine, which after a while e'en knock'd + him down dead drunk, so that he became moveless, though at first putting + in he struggled for a pretty while very much, till at last, certain bubbles + issuing out of his mouth, it ceased to move; this (because I had before found + them quickly to recover again, if they were taken out presently) I suffered + to lye above an hour in the Spirit; and after I had taken it out, and put + its body and legs into a natural posture, remained moveless about an hour; + but then, upon a sudden, as if it had been awaken out of a drunken sleep, + it suddenly reviv'd and ran away; being caught, and serv'd as before, he + for a while continued struggling and striving, till at last there issued + several bubbles out of its mouth, and then, tanquam animam expirasset, he + remained moveless for a good while ; but at length again recovering, it was + again redipt, and suffered to lye some hours in the Spirit; notwithstanding + which, after it had layen dry some three or four hours, it again recovered + life and motion: Which kind of Experiments, if prosecuted, which they highly + deserve, seem to me of no inconsiderable use towards the invention of the + Latent Scheme, (as the Noble Ve rulam calls it) or the hidden, unknown Texture of Bodies.
-Of what Figure this Creature appear'd through the Microscope, - the 32. Scheme (though not so carefully graven as it ought) will represent - to the eye, namely, That it had a large head A A, at the upper end of which - were two protuberant eyes, pearl'd like those of a Fly, but smaller B B; - of the Nose, or foremost part, issued two horns C C, of a shape sufficiently - differing from those of a blew Fly, though indeed they seem to be both the - same kind of Organ, and to serve for a kind of smelling; beyond these were - two indented jaws D D, which he open'd sideways, and was able to gape them - asunder very wide; and the ends of them being armed with teeth, which meeting - went between each other, it was able to grasp and hold a heavy body, three - or four times the bulk and weight of its own body: It had only six legs, - shap'd like those of a Fly, which, as I shewed before, is an Argument that - it is a winged Insect, and though I could not perceive any sign of them in - the middle part of its body (which seem'd to consist of three joints or pieces - E F G, out of which sprung two legs, yet 'tis known that there are of them +
Of what Figure this Creature appear'd through the Microscope, + the 32. Scheme (though not so carefully graven as it ought) will represent + to the eye, namely, That it had a large head A A, at the upper end of which + were two protuberant eyes, pearl'd like those of a Fly, but smaller B B; + of the Nose, or foremost part, issued two horns C C, of a shape sufficiently + differing from those of a blew Fly, though indeed they seem to be both the + same kind of Organ, and to serve for a kind of smelling; beyond these were + two indented jaws D D, which he open'd sideways, and was able to gape them + asunder very wide; and the ends of them being armed with teeth, which meeting + went between each other, it was able to grasp and hold a heavy body, three + or four times the bulk and weight of its own body: It had only six legs, + shap'd like those of a Fly, which, as I shewed before, is an Argument that + it is a winged Insect, and though I could not perceive any sign of them in + the middle part of its body (which seem'd to consist of three joints or pieces + E F G, out of which sprung two legs, yet 'tis known that there are of them that have long wings, and fly up and down in the air.
-The third and last part of its body I I I was bigger and - larger then the other two, unto which it was joyn'd by a very small middle, - and had a kind of loose shell, or another distinct part of its body H, which - seem'd to be interpos'd, and to keep the thorax and belly from touching. - The whole body was cas'd over with a very strong armour, and the belly I - I I was covered likewise with multitudes of small white shining brisles; - the legs, horns, head, and middle parts of its body were bestruck with hairs +
The third and last part of its body I I I was bigger and + larger then the other two, unto which it was joyn'd by a very small middle, + and had a kind of loose shell, or another distinct part of its body H, which + seem'd to be interpos'd, and to keep the thorax and belly from touching. + The whole body was cas'd over with a very strong armour, and the belly I + I I was covered likewise with multitudes of small white shining brisles; + the legs, horns, head, and middle parts of its body were bestruck with hairs also, but smaller and darker.
diff --git a/src/demo/share/jfc/SwingSet2/resources/bug.html b/src/demo/share/jfc/SwingSet2/resources/bug.html index bf3b159112f..d61cdb9ad25 100644 --- a/src/demo/share/jfc/SwingSet2/resources/bug.html +++ b/src/demo/share/jfc/SwingSet2/resources/bug.html @@ -1,7 +1,7 @@Untitled Document - + @@ -10,113 +10,113 @@Observ. LIV. Of a Louse.
--
-
This is a Creature so officious, that 'twill be known to - every one at one time or other, so busie, and so impudent, that it will - be intruding it self in every ones company, and so proud and aspiring withall, - that it fears not to trample on the best, and affects nothing so much as - a Crown; feeds and lives very high, and that makes it so saucy, as to pull - any one by the ears that comes in its way, and will never be quiet till - it has drawn blood: it is troubled at nothing so much as at a man that scratches - his head, as knowing that man is plotting and contriving some mischief against - it, and that makes it oftentime sculk into some meaner and lower place, and - run behind a mans back, though it go very much against the hair; which ill - conditions of it having made it better known then trusted, would exempt me - from making any further description of it, did not my faithful Mercury, my +
+
+
This is a Creature so officious, that 'twill be known to + every one at one time or other, so busie, and so impudent, that it will + be intruding it self in every ones company, and so proud and aspiring withall, + that it fears not to trample on the best, and affects nothing so much as + a Crown; feeds and lives very high, and that makes it so saucy, as to pull + any one by the ears that comes in its way, and will never be quiet till + it has drawn blood: it is troubled at nothing so much as at a man that scratches + his head, as knowing that man is plotting and contriving some mischief against + it, and that makes it oftentime sculk into some meaner and lower place, and + run behind a mans back, though it go very much against the hair; which ill + conditions of it having made it better known then trusted, would exempt me + from making any further description of it, did not my faithful Mercury, my Microscope, bring me other information of it.
-For this has discovered to me, by means of a very bright light - cast on it, that it is a Creature of a very odd shape ; it has a head shap'd - like that exprest in 35. Scheme marked with A, which seems almost Conical, - but is a little flatted on the upper and under sides, at the biggest part - of which, on either side behind the head (as it were, being the place where - other Creatures ears stand) are placed its two black shining goggle eyes - B B, looking backwards, and fenced round with several small cilia or hairs - that incompass it, so that it seems this Creature has no very good foresight: - It does not seem to have any eyelids, and therefore perhaps its eyes were - so placed, that it might the better cleanse them with its forelegs; and perhaps - this may be the reason, why they so much avoid and run from the light behind - them, for being made to live in the shady and dark recesses of the hair, - and thence probably their eye having a great aperture, the open and clear - light, especially that of the Sun, must needs very much offend them; to secure - these eyes from receiving any injury from the hairs through which it passes, - it has two horns that grow before it, in the place where one would have thought - the eyes should be; each of these C C have four joynts, which are fringed, - as 'twere, with small brisles, from which to the tip of its snout D, the - head seems very round and tapering, ending in a very sharp nose D, which - seems to have a small hole, and to be the passage through which he sucks +
For this has discovered to me, by means of a very bright light + cast on it, that it is a Creature of a very odd shape ; it has a head shap'd + like that exprest in 35. Scheme marked with A, which seems almost Conical, + but is a little flatted on the upper and under sides, at the biggest part + of which, on either side behind the head (as it were, being the place where + other Creatures ears stand) are placed its two black shining goggle eyes + B B, looking backwards, and fenced round with several small cilia or hairs + that incompass it, so that it seems this Creature has no very good foresight: + It does not seem to have any eyelids, and therefore perhaps its eyes were + so placed, that it might the better cleanse them with its forelegs; and perhaps + this may be the reason, why they so much avoid and run from the light behind + them, for being made to live in the shady and dark recesses of the hair, + and thence probably their eye having a great aperture, the open and clear + light, especially that of the Sun, must needs very much offend them; to secure + these eyes from receiving any injury from the hairs through which it passes, + it has two horns that grow before it, in the place where one would have thought + the eyes should be; each of these C C have four joynts, which are fringed, + as 'twere, with small brisles, from which to the tip of its snout D, the + head seems very round and tapering, ending in a very sharp nose D, which + seems to have a small hole, and to be the passage through which he sucks the blood.
-
Now whereas it if be plac'd on its back, with its belly - upwards, as it is in the 35. Scheme, it seems in several Positions to have - a resemblance of chaps, or jaws, as is represented in the Figure by E E, - yet in other postures those dark strokes disappear; and having kept several - of them in a box for two or three dayes, so that for all that time they had - nothing to feed on, I found, upon letting onecreep on my hand, that it immediately - fell to sucking, and did neither seem to thrust its nose very deep into the - skin, nor to open any kind of mouth, but I could plainly perceive a small - current of blood, which came directly from its snout, and past into its belly; - and about A there seem'd a contrivance, somewhat resembling a Pump, pair - of Bellows, or Heart, for by a very swift systole and diastole the blood +
Now whereas it if be plac'd on its back, with its belly + upwards, as it is in the 35. Scheme, it seems in several Positions to have + a resemblance of chaps, or jaws, as is represented in the Figure by E E, + yet in other postures those dark strokes disappear; and having kept several + of them in a box for two or three dayes, so that for all that time they had + nothing to feed on, I found, upon letting onecreep on my hand, that it immediately + fell to sucking, and did neither seem to thrust its nose very deep into the + skin, nor to open any kind of mouth, but I could plainly perceive a small + current of blood, which came directly from its snout, and past into its belly; + and about A there seem'd a contrivance, somewhat resembling a Pump, pair + of Bellows, or Heart, for by a very swift systole and diastole the blood seem'd drawn from the nose, and forced into the body.
-It did not seem at all, though I viewed it a good while as - it was sucking, to thrust more of its nose into the skin then the very snout - D, nor did it cause the least discernable pain, and yet the blood seem'd - to run through its head very quick and freely, so that it seems there is - no part of the skin but the blood is dispers'd into, nay, even into the - cuticula; for had it thrust its whole nose in from D to C C, it would not - have amounted to the supposed thickness of that tegument, the length of +
It did not seem at all, though I viewed it a good while as + it was sucking, to thrust more of its nose into the skin then the very snout + D, nor did it cause the least discernable pain, and yet the blood seem'd + to run through its head very quick and freely, so that it seems there is + no part of the skin but the blood is dispers'd into, nay, even into the + cuticula; for had it thrust its whole nose in from D to C C, it would not + have amounted to the supposed thickness of that tegument, the length of the nose being not more then a three hundredth part of an inch.
-It has six legs, covered with a very transparent shell, - and joynted exactly like a Crab's, or Lobster's; each leg is divided into - six parts by these joynts, and those have here and there several small hairs; - and at the end of each leg it has two claws, very properly adapted for its - peculiar use, being thereby inabled to walk very securely both on the skin - and hair; and indeed this contrivance of the feet is very curious, and could - not be made more commodiously and compendiously, for performing both these - requisite motions, of walking and climbing up the hair of a mans head, then - it is : for, by having the lesser claw (a) set so much short of the bigger - (b) when it walks on the skin the shorter touches not, and then the feet - are the same with those of a Mite, and several other small Insects, but by - means of the small joynts of the longer claw it can bend it round, and so - with both claws take hold of a hair, in the manner represented in the Figure, +
It has six legs, covered with a very transparent shell, + and joynted exactly like a Crab's, or Lobster's; each leg is divided into + six parts by these joynts, and those have here and there several small hairs; + and at the end of each leg it has two claws, very properly adapted for its + peculiar use, being thereby inabled to walk very securely both on the skin + and hair; and indeed this contrivance of the feet is very curious, and could + not be made more commodiously and compendiously, for performing both these + requisite motions, of walking and climbing up the hair of a mans head, then + it is : for, by having the lesser claw (a) set so much short of the bigger + (b) when it walks on the skin the shorter touches not, and then the feet + are the same with those of a Mite, and several other small Insects, but by + means of the small joynts of the longer claw it can bend it round, and so + with both claws take hold of a hair, in the manner represented in the Figure, the long transparent Cylinder F F F, being a Man's hair held by it.
-The Thorax seem'd cas'd with another kind of substance then - the belly, namely, with a thin transparent horny substance, which upon the fasting - of the Creature did not grow flaccid; through this I could plainly see the - blood, suck'd from my hand, to be variously distributed, and mov'd to and - fro; and about G there seem'd a pretty big white substance, which seem'd - to be moved within its thorax; besides, there appear'd very many small milk-white - vessels, which crost over the breast between the legs, out of which, on - either side, are many small branchings, these seem'd to be the veins and - arteries, for that which is analogus to blood in all Insects is milk-white. +
The Thorax seem'd cas'd with another kind of substance then + the belly, namely, with a thin transparent horny substance, which upon the fasting + of the Creature did not grow flaccid; through this I could plainly see the + blood, suck'd from my hand, to be variously distributed, and mov'd to and + fro; and about G there seem'd a pretty big white substance, which seem'd + to be moved within its thorax; besides, there appear'd very many small milk-white + vessels, which crost over the breast between the legs, out of which, on + either side, are many small branchings, these seem'd to be the veins and + arteries, for that which is analogus to blood in all Insects is milk-white.
-The belly is covered with a transparent substance likewise, - but more resembling a skin then a shell, for 'tis grain'd all over the belly - just like the skin in the palms of a man's hand, and when the belly is empty, - grows very flaccid and wrinkled ; at the upper end of this is placed the - stomach H H, and perhaps also the white spot I I may be the liver, or pancreas, - which by the peristaltick motion of the guts, is a little mov'd to and fro, - not with a systole and diastole, but rather with a thronging or justling +
The belly is covered with a transparent substance likewise, + but more resembling a skin then a shell, for 'tis grain'd all over the belly + just like the skin in the palms of a man's hand, and when the belly is empty, + grows very flaccid and wrinkled ; at the upper end of this is placed the + stomach H H, and perhaps also the white spot I I may be the liver, or pancreas, + which by the peristaltick motion of the guts, is a little mov'd to and fro, + not with a systole and diastole, but rather with a thronging or justling motion.
-Viewing one of these Creatures, after it had fasted two - dayes, all the hinder part was lank and flaccid, and the white spot I I - hardly mov'd, most of the white branchings disappear'd, and most also of - the redness or sucked blood in the guts, the peristaltick motion of which - was scarce discernable; but upon the suffering it to suck, it presently - fill'd the skin of the belly, and of the six scolop'd embosments on either side, - as full as it could be stuft ; the stomach and guts were as full as they - could hold; the peristaltick motion of the gut grew quick, and the justling - motion of I I accordingly ; multitudes of milk-white vessels seem'd quickly - filled, and turgid, which were perhaps the veins and arteries, and the Creature - was so greedy, that though it could not contain more, yet it continued sucking - as fast as ever, and as fast emptying it self behind : the digestion of this - Creature must needs be very quick, for though I perceiv'd the blood thicker - and blacker when suck'd, yet, when in the guts, it was of a very lovely - ruby colour, and that part of it, which was digested into the veins, seemed - white; whence it appears, that a further digestion of blood may make it - milk, at least of a resembling colour : What is else observable in the figure +
Viewing one of these Creatures, after it had fasted two + dayes, all the hinder part was lank and flaccid, and the white spot I I + hardly mov'd, most of the white branchings disappear'd, and most also of + the redness or sucked blood in the guts, the peristaltick motion of which + was scarce discernable; but upon the suffering it to suck, it presently + fill'd the skin of the belly, and of the six scolop'd embosments on either side, + as full as it could be stuft ; the stomach and guts were as full as they + could hold; the peristaltick motion of the gut grew quick, and the justling + motion of I I accordingly ; multitudes of milk-white vessels seem'd quickly + filled, and turgid, which were perhaps the veins and arteries, and the Creature + was so greedy, that though it could not contain more, yet it continued sucking + as fast as ever, and as fast emptying it self behind : the digestion of this + Creature must needs be very quick, for though I perceiv'd the blood thicker + and blacker when suck'd, yet, when in the guts, it was of a very lovely + ruby colour, and that part of it, which was digested into the veins, seemed + white; whence it appears, that a further digestion of blood may make it + milk, at least of a resembling colour : What is else observable in the figure of this Creature, maybe seen by the 35. Scheme.
diff --git a/src/demo/share/jfc/SwingSet2/resources/index.html b/src/demo/share/jfc/SwingSet2/resources/index.html index 0aa73c410ef..ba7d26068a1 100644 --- a/src/demo/share/jfc/SwingSet2/resources/index.html +++ b/src/demo/share/jfc/SwingSet2/resources/index.html @@ -1,7 +1,7 @@Untitled Document - + @@ -20,21 +20,21 @@Of a Louse
-
Images and text used by permission of Octavo +
Images and text used by permission of Octavo Corporation (www.octavo.com),
- (c) 1999 Octavo Corporation. All + (c) 1999 Octavo Corporation. All rights reserved.
- Octavo Corporation is a publisher of rare - books and manuscripts with digital tools and formats through partnerships - with libraries, museums, and individuals. Using high-resolution digital imaging - technology, Octavo releases digital rare books on CD-ROM as Adobe PDF files - which can be viewed on and printed from almost any computing platform. You - can view each page and the binding on your computer screen, zoom in to view - detail up to 800% in some cases, and search, copy and paste the "live" text - placed invisibly behind the page images which is available for selected Editions. - Also included in each edition is the work's collation and provenance, as well + Octavo Corporation is a publisher of rare + books and manuscripts with digital tools and formats through partnerships + with libraries, museums, and individuals. Using high-resolution digital imaging + technology, Octavo releases digital rare books on CD-ROM as Adobe PDF files + which can be viewed on and printed from almost any computing platform. You + can view each page and the binding on your computer screen, zoom in to view + detail up to 800% in some cases, and search, copy and paste the "live" text + placed invisibly behind the page images which is available for selected Editions. + Also included in each edition is the work's collation and provenance, as well as commentary by a noted expert in its field.
diff --git a/src/demo/share/jfc/SwingSet2/resources/king.html b/src/demo/share/jfc/SwingSet2/resources/king.html index a6f458af419..b14f2d84b51 100644 --- a/src/demo/share/jfc/SwingSet2/resources/king.html +++ b/src/demo/share/jfc/SwingSet2/resources/king.html @@ -1,34 +1,34 @@Untitled Document - + -
+
-Do here most humbly lay this small Present at Your - Majesties Royal feet. And though it comes accompany'd with two disadvantages, - the meanness of the Author, and of the Subject; yet in both I am incouraged +
Do here most humbly lay this small Present at Your + Majesties Royal feet. And though it comes accompany'd with two disadvantages, + the meanness of the Author, and of the Subject; yet in both I am incouraged by the greatness of your Mercy and your Knowledge. -
By the one I am taught , that you can forgive the most presumptuous - Offendors: And by the other, that you will not esteem the least work of Nature, +
By the one I am taught , that you can forgive the most presumptuous + Offendors: And by the other, that you will not esteem the least work of Nature, or Art, unworthy your Observation.
-Amidst the many felicities that have accompani'd your Majesties - happy Restauration and Government, it is none of the least considerable, that +
Amidst the many felicities that have accompani'd your Majesties + happy Restauration and Government, it is none of the least considerable, that Philosophy and Experimental Learning have prosper'd under your Royal Patronage.
-And as the calm prosperity of your Reign has given us the - leisure to follow these Studies of quiet and retirement, so it is just, that - the Fruits of them should, by way of acknowledgement, be return'd to your - Majesty. There are, Sir, several other of your Subjects, of your Royal Society, - now busie about Nobler matters: The Improvement of Manufactures and Agriculture, - the Increase of Commerce, the Advantage of Navigation: In all which they are +
And as the calm prosperity of your Reign has given us the + leisure to follow these Studies of quiet and retirement, so it is just, that + the Fruits of them should, by way of acknowledgement, be return'd to your + Majesty. There are, Sir, several other of your Subjects, of your Royal Society, + now busie about Nobler matters: The Improvement of Manufactures and Agriculture, + the Increase of Commerce, the Advantage of Navigation: In all which they are assisted by your Majesties Incouragement and Example.
-Amidst all those greater Designs, I here presume to bring - in that which is more proportionable to the smalness of my Abilities, and - to offer some of the least of all visible things, to that Mighty King, that - has establisht an Empire over the best of all Invisible things of this World, +
Amidst all those greater Designs, I here presume to bring + in that which is more proportionable to the smalness of my Abilities, and + to offer some of the least of all visible things, to that Mighty King, that + has establisht an Empire over the best of all Invisible things of this World, the Minds o f Men.
diff --git a/src/demo/share/jfc/SwingSet2/resources/preface.html b/src/demo/share/jfc/SwingSet2/resources/preface.html index 8138b288a18..b7c10d6eec1 100644 --- a/src/demo/share/jfc/SwingSet2/resources/preface.html +++ b/src/demo/share/jfc/SwingSet2/resources/preface.html @@ -1,7 +1,7 @@Untitled Document - + @@ -9,101 +9,101 @@THE PREFACE
-+-+
-It is the great prerogative of Mankind above - other Creatures, that we are not only able to behold the works of Nature, - or barely to sustein our lives by them, but we have also the power of considering, +
It is the great prerogative of Mankind above + other Creatures, that we are not only able to behold the works of Nature, + or barely to sustein our lives by them, but we have also the power of considering, comparing, altering, assisting, and improving them to various uses.
-And as this is the peculiar priviledge of humane Nature - in general, so is it capable of being so far advanced by the helps of Art, - and Experience, as to make some Men excel others in their Observations, +
And as this is the peculiar priviledge of humane Nature + in general, so is it capable of being so far advanced by the helps of Art, + and Experience, as to make some Men excel others in their Observations, and Deductions, almost as much as they do Beasts.
-By the addition of such artificial Instruments and methods, - there may be, in some manner, a reparation made for the mischiefs, and imperfection, - mankind has drawn upon itself, by negligence, and intemperance, and a wilful - and superstitious deserting the Prescripts and Rules of Nature, whereby - every man, both from a deriv'd corruption, innate and born with him, and - from his breeding and converse with men, is very subject to slip into all +
By the addition of such artificial Instruments and methods, + there may be, in some manner, a reparation made for the mischiefs, and imperfection, + mankind has drawn upon itself, by negligence, and intemperance, and a wilful + and superstitious deserting the Prescripts and Rules of Nature, whereby + every man, both from a deriv'd corruption, innate and born with him, and + from his breeding and converse with men, is very subject to slip into all sorts of errors.
-The only way which now remains for us to recover some degree - of those former perfections, seems to be, by rectifying the operations of - the Sense, the Memory, and Reason, since upon the evidence, the strength, - the integrity, and the right correspondence of all these, all the light, - by which our actions are to be guided, is to be renewed, and all our command +
The only way which now remains for us to recover some degree + of those former perfections, seems to be, by rectifying the operations of + the Sense, the Memory, and Reason, since upon the evidence, the strength, + the integrity, and the right correspondence of all these, all the light, + by which our actions are to be guided, is to be renewed, and all our command over things is to be establisht.
-It is therefore most worthy of our consideration, to recollect - their several defects, that so we may the better understand how to supply - them, and by what assistances we may inlarge their power, and secure them +
It is therefore most worthy of our consideration, to recollect + their several defects, that so we may the better understand how to supply + them, and by what assistances we may inlarge their power, and secure them in performing their particular duties.
-As for the actions of our Senses, we cannot but observe - them to be in many particulars much outdone by those of other Creatures, - and when at best, to be far short of the perfection they seem capable of - : And these infirmities of the Senses arise from a double cause, either - from the disproportion of the Object to the Organ, whereby an infinite number - of things can never enter into them, or else from error in the Perception, - that many things, which come within their reach, are not received in a right +
As for the actions of our Senses, we cannot but observe + them to be in many particulars much outdone by those of other Creatures, + and when at best, to be far short of the perfection they seem capable of + : And these infirmities of the Senses arise from a double cause, either + from the disproportion of the Object to the Organ, whereby an infinite number + of things can never enter into them, or else from error in the Perception, + that many things, which come within their reach, are not received in a right manner.
-The like frailties are to be found in the Memory; we often - let many things slip away from us, which deserve to be retain'd; and of - those which we treasure up, a great part is either frivolous or false ; - and if good, and substantial, either in tract of time obliterated, or at - best so overwhelmed and buried under more frothy notions, that when there +
The like frailties are to be found in the Memory; we often + let many things slip away from us, which deserve to be retain'd; and of + those which we treasure up, a great part is either frivolous or false ; + and if good, and substantial, either in tract of time obliterated, or at + best so overwhelmed and buried under more frothy notions, that when there is need of them, they are in vain sought for.
-The two main foundations being so deceivable, it is no wonder, - that all the succeeding works which we build upon them, of arguing, concluding, - defining, judging, and all the other degrees of Reason, are lyable to the - same imperfection, being, at best, either vain, or uncertain: So that the - errors of the understanding are answerable to the two other, being defective - both in the quantity and goodness of its knowledge; for the limits, to which - our thoughts are confind, are small in respect of the vast extent of Nature - it self; some parts of it are too large to be comprehended, and some too +
The two main foundations being so deceivable, it is no wonder, + that all the succeeding works which we build upon them, of arguing, concluding, + defining, judging, and all the other degrees of Reason, are lyable to the + same imperfection, being, at best, either vain, or uncertain: So that the + errors of the understanding are answerable to the two other, being defective + both in the quantity and goodness of its knowledge; for the limits, to which + our thoughts are confind, are small in respect of the vast extent of Nature + it self; some parts of it are too large to be comprehended, and some too little to be perceived.
-And from thence it must follow, that not having a full sensation - of the Object, we must be very lame and imperfect in our conceptions about - it, and in all the propositions which we build upon it; hence we often take - the shadow of things for the substance, small appearances for good similitudes, - similitudes for definitions; and even many of those, which we think to be - the most solid definitions, are rather expressions of our own misguided +
And from thence it must follow, that not having a full sensation + of the Object, we must be very lame and imperfect in our conceptions about + it, and in all the propositions which we build upon it; hence we often take + the shadow of things for the substance, small appearances for good similitudes, + similitudes for definitions; and even many of those, which we think to be + the most solid definitions, are rather expressions of our own misguided apprehensions then of the true nature of the things themselves.
-The effects of these imperfections are manifested in different - ways, according to the temper and disposition of the several minds of men, - some they incline to gross ignorance and stupidity, and others to a presumptuous - imposing on other mens Opinions, and a confident dogmatizing on matters, +
The effects of these imperfections are manifested in different + ways, according to the temper and disposition of the several minds of men, + some they incline to gross ignorance and stupidity, and others to a presumptuous + imposing on other mens Opinions, and a confident dogmatizing on matters, whereof there is no assurance to be given.
-Thus all the uncertainty, and mistakes of humane actions, - proceed either from the narrowness and wandring of our Senses, from the - slipperiness or delusion of our Memory, from the confinement or rashness - of our Understanding, so that 'tis no wonder, that our power over natural - causes and effects is so slowly improvd, seeing we are not only to contend - with the obscurity and difficulty of the things whereon we work and think, +
Thus all the uncertainty, and mistakes of humane actions, + proceed either from the narrowness and wandring of our Senses, from the + slipperiness or delusion of our Memory, from the confinement or rashness + of our Understanding, so that 'tis no wonder, that our power over natural + causes and effects is so slowly improvd, seeing we are not only to contend + with the obscurity and difficulty of the things whereon we work and think, but even the forces of our own minds conspire to betray us.
-These being the dangers in the process of humane Reason, - the remedies of them all can only proceed from the real, the mechanical, - the experimental Philosophy, which has this advantage over the Philosophy - of discourse and disputation, that whereas that chiefly aims at the subtilty - of its Deductions and Conclusions, without much regard to the first groundwork, - which ought to be well laid on the Sense and Memory ; so this intends the - right ordering of them all, and the making them serviceable to each other. +
These being the dangers in the process of humane Reason, + the remedies of them all can only proceed from the real, the mechanical, + the experimental Philosophy, which has this advantage over the Philosophy + of discourse and disputation, that whereas that chiefly aims at the subtilty + of its Deductions and Conclusions, without much regard to the first groundwork, + which ought to be well laid on the Sense and Memory ; so this intends the + right ordering of them all, and the making them serviceable to each other.
-The first thing to be undertaken in this weighty work, is - a watchfulness over the failings and an inlargement of the dominion, of - the Senses. To which end it is requisite, first, That there should be a - scrupulous choice, and a strict examination, of the reality, constancy, - and certainty of the Particulars that we admit: This is the first rise whereon - truth is to begin, and here the most severe, and most impartial diligence, - must be imployed ; the storing up of all, without any regard to evidence +
The first thing to be undertaken in this weighty work, is + a watchfulness over the failings and an inlargement of the dominion, of + the Senses. To which end it is requisite, first, That there should be a + scrupulous choice, and a strict examination, of the reality, constancy, + and certainty of the Particulars that we admit: This is the first rise whereon + truth is to begin, and here the most severe, and most impartial diligence, + must be imployed ; the storing up of all, without any regard to evidence or use, will only tend to darkness and confusion.
-We must not therefore esteem the riches of our Philosophical - treasure by the number only, but chiefly by the weight; the most vulgar - Instances are not to be neglected, but above all, the most instructive are - to be entertain'd: the footsteps of Nature are to be trac'd, not only in - her ordinary course,but when she seems to be put to her shifts, to make - many doublings and turnings, and to use some kind of art in indeavouring +
We must not therefore esteem the riches of our Philosophical + treasure by the number only, but chiefly by the weight; the most vulgar + Instances are not to be neglected, but above all, the most instructive are + to be entertain'd: the footsteps of Nature are to be trac'd, not only in + her ordinary course,but when she seems to be put to her shifts, to make + many doublings and turnings, and to use some kind of art in indeavouring to avoid our discovery.
diff --git a/src/demo/share/jfc/SwingSet2/resources/seaweed.html b/src/demo/share/jfc/SwingSet2/resources/seaweed.html index fd2a9433630..f386751dd7c 100644 --- a/src/demo/share/jfc/SwingSet2/resources/seaweed.html +++ b/src/demo/share/jfc/SwingSet2/resources/seaweed.html @@ -1,7 +1,7 @@
Untitled Document - + @@ -11,47 +11,47 @@
-
For curiosity and beauty, I have not among all the Plants - or Vegetables I have yet observ'd, seen any one comparable to this Sea-weed - I have here describ'd, of which I am able to say very little more then what - is represented by the second Figure of the ninth Scheme: Namely, that it is - a Plant which grows upon the Rocks under the water, and increases and spreads - it self into a great tuft, which is not onely handsomely branch'd into several - leaves, but the whole surface of the Plant is cover'd over with a most curious - kind of carv'd work, which consists of a texture much resembling a Honeycomb; - for the whole surface on both sides is cover'd over with a multitude of very - small holes, being no bigger then so many holes made with the point of a small - Pinn, and rang'd in the neatest and most delicate order imaginable, they being - plac'd in the manner of a Quincunx, or very much like the rows of the eyes - of a Fly, the rows or orders being very regular, which way soever they are - observ'd: what the texture was, as it appear'd through a pretty bigg Magnifying - Microscope, I have here adjoin'd in the first Figure of the 14. Scheme. which - round Area A B C D represents a part of the surface about one eighth part - of an Inch in Diameter: Those little holes, which to the eye look'd round, - like so many little spots, here appear'd very regularly shap'd holes, representing - almost the shape of the sole of a round toed shoe, the hinder part of which, - is, as it were, trod on or cover'd by the toe of that next below it; these - holes seem'd wall'd about with a very thin and transparent substance, looking - of a pale straw-colour; from the edge of which, against the middle of each - hole, were sprouted out four small transparent straw-colour'd Thorns, which - seem'd to protect and cover those cavities, from either side two; neer the - root of this Plant, were sprouted out several small branches of a kind of +
For curiosity and beauty, I have not among all the Plants + or Vegetables I have yet observ'd, seen any one comparable to this Sea-weed + I have here describ'd, of which I am able to say very little more then what + is represented by the second Figure of the ninth Scheme: Namely, that it is + a Plant which grows upon the Rocks under the water, and increases and spreads + it self into a great tuft, which is not onely handsomely branch'd into several + leaves, but the whole surface of the Plant is cover'd over with a most curious + kind of carv'd work, which consists of a texture much resembling a Honeycomb; + for the whole surface on both sides is cover'd over with a multitude of very + small holes, being no bigger then so many holes made with the point of a small + Pinn, and rang'd in the neatest and most delicate order imaginable, they being + plac'd in the manner of a Quincunx, or very much like the rows of the eyes + of a Fly, the rows or orders being very regular, which way soever they are + observ'd: what the texture was, as it appear'd through a pretty bigg Magnifying + Microscope, I have here adjoin'd in the first Figure of the 14. Scheme. which + round Area A B C D represents a part of the surface about one eighth part + of an Inch in Diameter: Those little holes, which to the eye look'd round, + like so many little spots, here appear'd very regularly shap'd holes, representing + almost the shape of the sole of a round toed shoe, the hinder part of which, + is, as it were, trod on or cover'd by the toe of that next below it; these + holes seem'd wall'd about with a very thin and transparent substance, looking + of a pale straw-colour; from the edge of which, against the middle of each + hole, were sprouted out four small transparent straw-colour'd Thorns, which + seem'd to protect and cover those cavities, from either side two; neer the + root of this Plant, were sprouted out several small branches of a kind of bastard Coralline, curiously branch'd, though small.
-And to confirm this, having lately the opportunity of viewing - the large Plant (if I may so call it) of a Sponge petrify'd, of which I made - mention in the last Observation, I found, that each of the Branches or Figures - of it, did, by the range of its pores, exhibit just such a texture, the rows - of pores crossing one another, much after the manner as the rows of eyes do - which are describ'd in the 26. Scheme : Coralline also, and several sorts of - white Coral, I have with a Microscope observ'd very curiously shap'd. And - I doubt not, but that he that shall observe these several kinds of Plants that - grow upon Rocks, which the Sea sometimes overflows, and those heaps of others - which are vomited out of it upon the shore, may find multitudes of little - Plants, and other bodies, which like this will afford very beautifull objects - for the Microscope ; and this Specimen here is adjoin'd onely to excite their - curiosities who have opportunity of observing to examine and collect what - they find worthy their notice; for the Sea, among terrestrial bodies, is also - a prolifick mother, and affords as many Instances of spontaneous generations +
And to confirm this, having lately the opportunity of viewing + the large Plant (if I may so call it) of a Sponge petrify'd, of which I made + mention in the last Observation, I found, that each of the Branches or Figures + of it, did, by the range of its pores, exhibit just such a texture, the rows + of pores crossing one another, much after the manner as the rows of eyes do + which are describ'd in the 26. Scheme : Coralline also, and several sorts of + white Coral, I have with a Microscope observ'd very curiously shap'd. And + I doubt not, but that he that shall observe these several kinds of Plants that + grow upon Rocks, which the Sea sometimes overflows, and those heaps of others + which are vomited out of it upon the shore, may find multitudes of little + Plants, and other bodies, which like this will afford very beautifull objects + for the Microscope ; and this Specimen here is adjoin'd onely to excite their + curiosities who have opportunity of observing to examine and collect what + they find worthy their notice; for the Sea, among terrestrial bodies, is also + a prolifick mother, and affords as many Instances of spontaneous generations as either the Air or Earth.
diff --git a/src/demo/share/jfc/SwingSet2/resources/title.html b/src/demo/share/jfc/SwingSet2/resources/title.html index ea13f5cf005..0e99558ccd7 100644 --- a/src/demo/share/jfc/SwingSet2/resources/title.html +++ b/src/demo/share/jfc/SwingSet2/resources/title.html @@ -1,7 +1,7 @@Untitled Document - + @@ -14,18 +14,18 @@
MADE BY
MAGNIFYING GLASSES.
WITH
-OBSERVATIONS and INQUIRIES +
OBSERVATIONS and INQUIRIES thereupon.
-By R. HOOKE +
By R. HOOKE , Fellow of the ROYAL SOCIETY .
-
-
-
-
LONDON, Printed by Jo. - Martyn, and Ja. Allestry, - Printers to the ROYAL SOCIETY , and are to - be sold at their Shop at the Bell in S. Paul's Church-yard. M +
+
+
+
LONDON, Printed by Jo. + Martyn, and Ja. Allestry, + Printers to the ROYAL SOCIETY , and are to + be sold at their Shop at the Bell in S. Paul's Church-yard. M D C L X V.
diff --git a/src/hotspot/.editorconfig b/src/hotspot/.editorconfig deleted file mode 100644 index 48e63362b54..00000000000 --- a/src/hotspot/.editorconfig +++ /dev/null @@ -1,3 +0,0 @@ -[*.{cpp,hpp,c,h}] -indent_style = space -indent_size = 2 diff --git a/src/hotspot/cpu/aarch64/aarch64.ad b/src/hotspot/cpu/aarch64/aarch64.ad index 76e3c92ddc2..f367362b4d8 100644 --- a/src/hotspot/cpu/aarch64/aarch64.ad +++ b/src/hotspot/cpu/aarch64/aarch64.ad @@ -1,5 +1,5 @@ // -// Copyright (c) 2003, 2024, Oracle and/or its affiliates. All rights reserved. +// Copyright (c) 2003, 2025, Oracle and/or its affiliates. All rights reserved. // Copyright (c) 2014, 2024, Red Hat, Inc. All rights reserved. // DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. // @@ -2296,6 +2296,26 @@ bool Matcher::match_rule_supported(int opcode) { return false; } break; + case Op_FmaHF: + // UseFMA flag also needs to be checked along with FEAT_FP16 + if (!UseFMA || !is_feat_fp16_supported()) { + return false; + } + break; + case Op_AddHF: + case Op_SubHF: + case Op_MulHF: + case Op_DivHF: + case Op_MinHF: + case Op_MaxHF: + case Op_SqrtHF: + // Half-precision floating point scalar operations require FEAT_FP16 + // to be available. FEAT_FP16 is enabled if both "fphp" and "asimdhp" + // features are supported. + if (!is_feat_fp16_supported()) { + return false; + } + break; } return true; // Per default match rules are supported. @@ -2306,11 +2326,11 @@ const RegMask* Matcher::predicate_reg_mask(void) { } bool Matcher::supports_vector_calling_convention(void) { - return EnableVectorSupport && UseVectorStubs; + return EnableVectorSupport; } OptoRegPair Matcher::vector_return_value(uint ideal_reg) { - assert(EnableVectorSupport && UseVectorStubs, "sanity"); + assert(EnableVectorSupport, "sanity"); int lo = V0_num; int hi = V0_H_num; if (ideal_reg == Op_VecX || ideal_reg == Op_VecA) { @@ -4599,6 +4619,15 @@ operand immF0() interface(CONST_INTER); %} +// Half Float (FP16) Immediate +operand immH() +%{ + match(ConH); + op_cost(0); + format %{ %} + interface(CONST_INTER); +%} + // operand immFPacked() %{ @@ -6942,6 +6971,21 @@ instruct loadConD(vRegD dst, immD con) %{ ins_pipe(fp_load_constant_d); %} +// Load Half Float Constant +// The "ldr" instruction loads a 32-bit word from the constant pool into a +// 32-bit register but only the bottom half will be populated and the top +// 16 bits are zero. +instruct loadConH(vRegF dst, immH con) %{ + match(Set dst con); + format %{ + "ldrs $dst, [$constantaddress]\t# load from constant table: half float=$con\n\t" + %} + ins_encode %{ + __ ldrs(as_FloatRegister($dst$$reg), $constantaddress($con)); + %} + ins_pipe(fp_load_constant_s); +%} + // Store Instructions // Store Byte @@ -7717,14 +7761,12 @@ instruct popCountI(iRegINoSp dst, iRegIorL2I src, vRegF tmp) %{ effect(TEMP tmp); ins_cost(INSN_COST * 13); - format %{ "movw $src, $src\n\t" - "mov $tmp, $src\t# vector (1D)\n\t" + format %{ "fmovs $tmp, $src\t# vector (1S)\n\t" "cnt $tmp, $tmp\t# vector (8B)\n\t" "addv $tmp, $tmp\t# vector (8B)\n\t" "mov $dst, $tmp\t# vector (1D)" %} ins_encode %{ - __ movw($src$$Register, $src$$Register); // ensure top 32 bits 0 - __ mov($tmp$$FloatRegister, __ D, 0, $src$$Register); + __ fmovs($tmp$$FloatRegister, $src$$Register); __ cnt($tmp$$FloatRegister, __ T8B, $tmp$$FloatRegister); __ addv($tmp$$FloatRegister, __ T8B, $tmp$$FloatRegister); __ mov($dst$$Register, $tmp$$FloatRegister, __ D, 0); @@ -8144,6 +8186,7 @@ instruct castPP(iRegPNoSp dst) instruct castII(iRegI dst) %{ + predicate(VerifyConstraintCasts == 0); match(Set dst (CastII dst)); size(0); @@ -8153,8 +8196,22 @@ instruct castII(iRegI dst) ins_pipe(pipe_class_empty); %} +instruct castII_checked(iRegI dst, rFlagsReg cr) +%{ + predicate(VerifyConstraintCasts > 0); + match(Set dst (CastII dst)); + effect(KILL cr); + + format %{ "# castII_checked of $dst" %} + ins_encode %{ + __ verify_int_in_range(_idx, bottom_type()->is_int(), $dst$$Register, rscratch1); + %} + ins_pipe(pipe_slow); +%} + instruct castLL(iRegL dst) %{ + predicate(VerifyConstraintCasts == 0); match(Set dst (CastLL dst)); size(0); @@ -8164,6 +8221,29 @@ instruct castLL(iRegL dst) ins_pipe(pipe_class_empty); %} +instruct castLL_checked(iRegL dst, rFlagsReg cr) +%{ + predicate(VerifyConstraintCasts > 0); + match(Set dst (CastLL dst)); + effect(KILL cr); + + format %{ "# castLL_checked of $dst" %} + ins_encode %{ + __ verify_long_in_range(_idx, bottom_type()->is_long(), $dst$$Register, rscratch1); + %} + ins_pipe(pipe_slow); +%} + +instruct castHH(vRegF dst) +%{ + match(Set dst (CastHH dst)); + size(0); + format %{ "# castHH of $dst" %} + ins_encode(/* empty encoding */); + ins_cost(0); + ins_pipe(pipe_class_empty); +%} + instruct castFF(vRegF dst) %{ match(Set dst (CastFF dst)); @@ -13606,6 +13686,17 @@ instruct bits_reverse_L(iRegLNoSp dst, iRegL src) // ============================================================================ // Floating Point Arithmetic Instructions +instruct addHF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ + match(Set dst (AddHF src1 src2)); + format %{ "faddh $dst, $src1, $src2" %} + ins_encode %{ + __ faddh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister); + %} + ins_pipe(fp_dop_reg_reg_s); +%} + instruct addF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ match(Set dst (AddF src1 src2)); @@ -13636,6 +13727,17 @@ instruct addD_reg_reg(vRegD dst, vRegD src1, vRegD src2) %{ ins_pipe(fp_dop_reg_reg_d); %} +instruct subHF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ + match(Set dst (SubHF src1 src2)); + format %{ "fsubh $dst, $src1, $src2" %} + ins_encode %{ + __ fsubh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister); + %} + ins_pipe(fp_dop_reg_reg_s); +%} + instruct subF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ match(Set dst (SubF src1 src2)); @@ -13666,6 +13768,17 @@ instruct subD_reg_reg(vRegD dst, vRegD src1, vRegD src2) %{ ins_pipe(fp_dop_reg_reg_d); %} +instruct mulHF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ + match(Set dst (MulHF src1 src2)); + format %{ "fmulh $dst, $src1, $src2" %} + ins_encode %{ + __ fmulh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister); + %} + ins_pipe(fp_dop_reg_reg_s); +%} + instruct mulF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ match(Set dst (MulF src1 src2)); @@ -13696,6 +13809,20 @@ instruct mulD_reg_reg(vRegD dst, vRegD src1, vRegD src2) %{ ins_pipe(fp_dop_reg_reg_d); %} +// src1 * src2 + src3 (half-precision float) +instruct maddHF_reg_reg(vRegF dst, vRegF src1, vRegF src2, vRegF src3) %{ + match(Set dst (FmaHF src3 (Binary src1 src2))); + format %{ "fmaddh $dst, $src1, $src2, $src3" %} + ins_encode %{ + assert(UseFMA, "Needs FMA instructions support."); + __ fmaddh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister, + $src3$$FloatRegister); + %} + ins_pipe(pipe_class_default); +%} + // src1 * src2 + src3 instruct maddF_reg_reg(vRegF dst, vRegF src1, vRegF src2, vRegF src3) %{ match(Set dst (FmaF src3 (Binary src1 src2))); @@ -13837,6 +13964,29 @@ instruct mnsubD_reg_reg(vRegD dst, vRegD src1, vRegD src2, vRegD src3, immD0 zer ins_pipe(pipe_class_default); %} +// Math.max(HH)H (half-precision float) +instruct maxHF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ + match(Set dst (MaxHF src1 src2)); + format %{ "fmaxh $dst, $src1, $src2" %} + ins_encode %{ + __ fmaxh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister); + %} + ins_pipe(fp_dop_reg_reg_s); +%} + +// Math.min(HH)H (half-precision float) +instruct minHF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ + match(Set dst (MinHF src1 src2)); + format %{ "fminh $dst, $src1, $src2" %} + ins_encode %{ + __ fminh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister); + %} + ins_pipe(fp_dop_reg_reg_s); +%} // Math.max(FF)F instruct maxF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ @@ -13894,6 +14044,16 @@ instruct minD_reg_reg(vRegD dst, vRegD src1, vRegD src2) %{ ins_pipe(fp_dop_reg_reg_d); %} +instruct divHF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ + match(Set dst (DivHF src1 src2)); + format %{ "fdivh $dst, $src1, $src2" %} + ins_encode %{ + __ fdivh($dst$$FloatRegister, + $src1$$FloatRegister, + $src2$$FloatRegister); + %} + ins_pipe(fp_div_s); +%} instruct divF_reg_reg(vRegF dst, vRegF src1, vRegF src2) %{ match(Set dst (DivF src1 src2)); @@ -14067,6 +14227,16 @@ instruct sqrtF_reg(vRegF dst, vRegF src) %{ ins_pipe(fp_div_d); %} +instruct sqrtHF_reg(vRegF dst, vRegF src) %{ + match(Set dst (SqrtHF src)); + format %{ "fsqrth $dst, $src" %} + ins_encode %{ + __ fsqrth($dst$$FloatRegister, + $src$$FloatRegister); + %} + ins_pipe(fp_div_s); +%} + // Math.rint, floor, ceil instruct roundD_reg(vRegD dst, vRegD src, immI rmode) %{ match(Set dst (RoundDoubleMode src rmode)); @@ -17116,6 +17286,64 @@ instruct expandBitsL_memcon(iRegINoSp dst, memory8 mem, immL mask, ins_pipe(pipe_slow); %} +//----------------------------- Reinterpret ---------------------------------- +// Reinterpret a half-precision float value in a floating point register to a general purpose register +instruct reinterpretHF2S(iRegINoSp dst, vRegF src) %{ + match(Set dst (ReinterpretHF2S src)); + format %{ "reinterpretHF2S $dst, $src" %} + ins_encode %{ + __ smov($dst$$Register, $src$$FloatRegister, __ H, 0); + %} + ins_pipe(pipe_slow); +%} + +// Reinterpret a half-precision float value in a general purpose register to a floating point register +instruct reinterpretS2HF(vRegF dst, iRegINoSp src) %{ + match(Set dst (ReinterpretS2HF src)); + format %{ "reinterpretS2HF $dst, $src" %} + ins_encode %{ + __ mov($dst$$FloatRegister, __ H, 0, $src$$Register); + %} + ins_pipe(pipe_slow); +%} + +// Without this optimization, ReinterpretS2HF (ConvF2HF src) would result in the following +// instructions (the first two are for ConvF2HF and the last instruction is for ReinterpretS2HF) - +// fcvt $tmp1_fpr, $src_fpr // Convert float to half-precision float +// mov $tmp2_gpr, $tmp1_fpr // Move half-precision float in FPR to a GPR +// mov $dst_fpr, $tmp2_gpr // Move the result from a GPR to an FPR +// The move from FPR to GPR in ConvF2HF and the move from GPR to FPR in ReinterpretS2HF +// can be omitted in this pattern, resulting in - +// fcvt $dst, $src // Convert float to half-precision float +instruct convF2HFAndS2HF(vRegF dst, vRegF src) +%{ + match(Set dst (ReinterpretS2HF (ConvF2HF src))); + format %{ "convF2HFAndS2HF $dst, $src" %} + ins_encode %{ + __ fcvtsh($dst$$FloatRegister, $src$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + +// Without this optimization, ConvHF2F (ReinterpretHF2S src) would result in the following +// instructions (the first one is for ReinterpretHF2S and the last two are for ConvHF2F) - +// mov $tmp1_gpr, $src_fpr // Move the half-precision float from an FPR to a GPR +// mov $tmp2_fpr, $tmp1_gpr // Move the same value from GPR to an FPR +// fcvt $dst_fpr, $tmp2_fpr // Convert the half-precision float to 32-bit float +// The move from FPR to GPR in ReinterpretHF2S and the move from GPR to FPR in ConvHF2F +// can be omitted as the input (src) is already in an FPR required for the fcvths instruction +// resulting in - +// fcvt $dst, $src // Convert half-precision float to a 32-bit float +instruct convHF2SAndHF2F(vRegF dst, vRegF src) +%{ + match(Set dst (ConvHF2F (ReinterpretHF2S src))); + format %{ "convHF2SAndHF2F $dst, $src" %} + ins_encode %{ + __ fcvths($dst$$FloatRegister, $src$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + // ============================================================================ // This name is KNOWN by the ADLC and cannot be changed. // The ADLC forces a 'TypeRawPtr::BOTTOM' output type diff --git a/src/hotspot/cpu/aarch64/aarch64_vector.ad b/src/hotspot/cpu/aarch64/aarch64_vector.ad index c7a0fc5724b..b4e6d79347f 100644 --- a/src/hotspot/cpu/aarch64/aarch64_vector.ad +++ b/src/hotspot/cpu/aarch64/aarch64_vector.ad @@ -1,6 +1,6 @@ // // Copyright (c) 2020, 2025, Oracle and/or its affiliates. All rights reserved. -// Copyright (c) 2020, 2024, Arm Limited. All rights reserved. +// Copyright (c) 2020, 2025, Arm Limited. All rights reserved. // DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. // // This code is free software; you can redistribute it and/or modify it @@ -225,6 +225,26 @@ source %{ return false; } break; + case Op_AddVHF: + case Op_SubVHF: + case Op_MulVHF: + case Op_DivVHF: + case Op_MinVHF: + case Op_MaxVHF: + case Op_SqrtVHF: + // FEAT_FP16 is enabled if both "fphp" and "asimdhp" features are supported. + // Only the Neon instructions need this check. SVE supports half-precision floats + // by default. + if (UseSVE == 0 && !is_feat_fp16_supported()) { + return false; + } + break; + case Op_FmaVHF: + // UseFMA flag needs to be checked along with FEAT_FP16 + if (!UseFMA || (UseSVE == 0 && !is_feat_fp16_supported())) { + return false; + } + break; default: break; } @@ -270,6 +290,19 @@ source %{ case Op_StoreVectorScatter: opcode = Op_StoreVectorScatterMasked; break; + // Currently, the masked versions of the following 8 Float16 operations are disabled. + // When the support for Float16 vector classes is added in VectorAPI and the masked + // Float16 IR can be generated, these masked operations will be enabled and relevant + // backend support added. + case Op_AddVHF: + case Op_SubVHF: + case Op_MulVHF: + case Op_DivVHF: + case Op_MaxVHF: + case Op_MinVHF: + case Op_SqrtVHF: + case Op_FmaVHF: + return false; default: break; } @@ -583,6 +616,22 @@ instruct vaddL(vReg dst, vReg src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vaddHF(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (AddVHF src1 src2)); + format %{ "vaddHF $dst, $src1, $src2" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ fadd($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + } else { + assert(UseSVE > 0, "must be sve"); + __ sve_fadd($dst$$FloatRegister, __ H, $src1$$FloatRegister, $src2$$FloatRegister); + } + %} + ins_pipe(pipe_slow); +%} + instruct vaddF(vReg dst, vReg src1, vReg src2) %{ match(Set dst (AddVF src1 src2)); format %{ "vaddF $dst, $src1, $src2" %} @@ -807,6 +856,22 @@ instruct vsubL(vReg dst, vReg src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vsubHF(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (SubVHF src1 src2)); + format %{ "vsubHF $dst, $src1, $src2" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ fsub($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + } else { + assert(UseSVE > 0, "must be sve"); + __ sve_fsub($dst$$FloatRegister, __ H, $src1$$FloatRegister, $src2$$FloatRegister); + } + %} + ins_pipe(pipe_slow); +%} + instruct vsubF(vReg dst, vReg src1, vReg src2) %{ match(Set dst (SubVF src1 src2)); format %{ "vsubF $dst, $src1, $src2" %} @@ -1004,6 +1069,22 @@ instruct vmulL_sve(vReg dst_src1, vReg src2) %{ // vector mul - floating-point +instruct vmulHF(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (MulVHF src1 src2)); + format %{ "vmulHF $dst, $src1, $src2" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ fmul($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + } else { + assert(UseSVE > 0, "must be sve"); + __ sve_fmul($dst$$FloatRegister, __ H, $src1$$FloatRegister, $src2$$FloatRegister); + } + %} + ins_pipe(pipe_slow); +%} + instruct vmulF(vReg dst, vReg src1, vReg src2) %{ match(Set dst (MulVF src1 src2)); format %{ "vmulF $dst, $src1, $src2" %} @@ -1102,6 +1183,28 @@ instruct vmulD_masked(vReg dst_src1, vReg src2, pRegGov pg) %{ // vector float div +instruct vdivHF_neon(vReg dst, vReg src1, vReg src2) %{ + predicate(VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst (DivVHF src1 src2)); + format %{ "vdivHF_neon $dst, $src1, $src2" %} + ins_encode %{ + __ fdiv($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + +instruct vdivHF_sve(vReg dst_src1, vReg src2) %{ + predicate(!VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst_src1 (DivVHF dst_src1 src2)); + format %{ "vdivHF_sve $dst_src1, $dst_src1, $src2" %} + ins_encode %{ + assert(UseSVE > 0, "must be sve"); + __ sve_fdiv($dst_src1$$FloatRegister, __ H, ptrue, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + instruct vdivF_neon(vReg dst, vReg src1, vReg src2) %{ predicate(VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); match(Set dst (DivVF src1 src2)); @@ -1991,6 +2094,21 @@ instruct vnegD_masked(vReg dst_src, pRegGov pg) %{ // vector sqrt +instruct vsqrtHF(vReg dst, vReg src) %{ + match(Set dst (SqrtVHF src)); + format %{ "vsqrtHF $dst, $src" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ fsqrt($dst$$FloatRegister, get_arrangement(this), $src$$FloatRegister); + } else { + assert(UseSVE > 0, "must be sve"); + __ sve_fsqrt($dst$$FloatRegister, __ H, ptrue, $src$$FloatRegister); + } + %} + ins_pipe(pipe_slow); +%} + instruct vsqrtF(vReg dst, vReg src) %{ match(Set dst (SqrtVF src)); format %{ "vsqrtF $dst, $src" %} @@ -2069,7 +2187,7 @@ instruct vminL_sve(vReg dst_src1, vReg src2) %{ ins_pipe(pipe_slow); %} -// vector min - B/S/I/F/D +// vector min - B/S/I/HF/F/D instruct vmin_neon(vReg dst, vReg src1, vReg src2) %{ predicate(Matcher::vector_element_basic_type(n) != T_LONG && @@ -2110,6 +2228,29 @@ instruct vmin_sve(vReg dst_src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vmin_HF_neon(vReg dst, vReg src1, vReg src2) %{ + predicate(VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst (MinVHF src1 src2)); + format %{ "vmin_HF_neon $dst, $src1, $src2\t# Half float" %} + ins_encode %{ + __ fmin($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + +instruct vmin_HF_sve(vReg dst_src1, vReg src2) %{ + predicate(!VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst_src1 (MinVHF dst_src1 src2)); + format %{ "vmin_HF_sve $dst_src1, $dst_src1, $src2\t# Half float" %} + ins_encode %{ + assert(UseSVE > 0, "must be sve"); + __ sve_fmin($dst_src1$$FloatRegister, __ H, + ptrue, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + // vector min - predicated instruct vmin_masked(vReg dst_src1, vReg src2, pRegGov pg) %{ @@ -2226,7 +2367,7 @@ instruct vmaxL_sve(vReg dst_src1, vReg src2) %{ ins_pipe(pipe_slow); %} -// vector max - B/S/I/F/D +// vector max - B/S/I/HF/F/D instruct vmax_neon(vReg dst, vReg src1, vReg src2) %{ predicate(Matcher::vector_element_basic_type(n) != T_LONG && @@ -2267,6 +2408,29 @@ instruct vmax_sve(vReg dst_src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vmax_HF_neon(vReg dst, vReg src1, vReg src2) %{ + predicate(VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst (MaxVHF src1 src2)); + format %{ "vmax_HF_neon $dst, $src1, $src2\t# Half float" %} + ins_encode %{ + __ fmax($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + +instruct vmax_HF_sve(vReg dst_src1, vReg src2) %{ + predicate(!VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst_src1 (MaxVHF dst_src1 src2)); + format %{ "vmax_HF_sve $dst_src1, $dst_src1, $src2\t# Half float" %} + ins_encode %{ + assert(UseSVE > 0, "must be sve"); + __ sve_fmax($dst_src1$$FloatRegister, __ H, + ptrue, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + // vector max - predicated instruct vmax_masked(vReg dst_src1, vReg src2, pRegGov pg) %{ @@ -2413,8 +2577,9 @@ instruct vmla_masked(vReg dst_src1, vReg src2, vReg src3, pRegGov pg) %{ // dst_src1 = src2 * src3 + dst_src1 instruct vfmla(vReg dst_src1, vReg src2, vReg src3) %{ - match(Set dst_src1 (FmaVF dst_src1 (Binary src2 src3))); - match(Set dst_src1 (FmaVD dst_src1 (Binary src2 src3))); + match(Set dst_src1 (FmaVHF dst_src1 (Binary src2 src3))); + match(Set dst_src1 (FmaVF dst_src1 (Binary src2 src3))); + match(Set dst_src1 (FmaVD dst_src1 (Binary src2 src3))); format %{ "vfmla $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -4613,6 +4778,23 @@ instruct replicateD(vReg dst, vRegD src) %{ ins_pipe(pipe_slow); %} +// Replicate a half-precision float value held in a floating point register +instruct replicateHF(vReg dst, vRegF src) %{ + predicate(Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst (Replicate src)); + format %{ "replicateHF $dst, $src\t# replicate half-precision float" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ dup($dst$$FloatRegister, get_arrangement(this), $src$$FloatRegister); + } else { // length_in_bytes must be > 16 and SVE should be enabled + assert(UseSVE > 0, "must be sve"); + __ sve_cpy($dst$$FloatRegister, __ H, ptrue, $src$$FloatRegister); + } + %} + ins_pipe(pipe_slow); +%} + // replicate from imm instruct replicateI_imm_le128b(vReg dst, immI con) %{ @@ -4681,6 +4863,23 @@ instruct replicateL_imm8_gt128b(vReg dst, immL8_shift8 con) %{ ins_pipe(pipe_slow); %} +// Replicate a 16-bit half precision float value +instruct replicateHF_imm(vReg dst, immH con) %{ + match(Set dst (Replicate con)); + format %{ "replicateHF_imm $dst, $con\t# replicate immediate half-precision float" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + int imm = (int)($con$$constant) & 0xffff; + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ mov($dst$$FloatRegister, get_arrangement(this), imm); + } else { // length_in_bytes must be > 16 and SVE should be enabled + assert(UseSVE > 0, "must be sve"); + __ sve_dup($dst$$FloatRegister, __ H, imm); + } + %} + ins_pipe(pipe_slow); +%} + // ------------------------------ Vector insert -------------------------------- // BYTE, SHORT, INT diff --git a/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4 b/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4 index 171bc390545..cc07e0e4076 100644 --- a/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4 +++ b/src/hotspot/cpu/aarch64/aarch64_vector_ad.m4 @@ -1,6 +1,6 @@ // // Copyright (c) 2020, 2025, Oracle and/or its affiliates. All rights reserved. -// Copyright (c) 2020, 2024, Arm Limited. All rights reserved. +// Copyright (c) 2020, 2025, Arm Limited. All rights reserved. // DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. // // This code is free software; you can redistribute it and/or modify it @@ -215,6 +215,26 @@ source %{ return false; } break; + case Op_AddVHF: + case Op_SubVHF: + case Op_MulVHF: + case Op_DivVHF: + case Op_MinVHF: + case Op_MaxVHF: + case Op_SqrtVHF: + // FEAT_FP16 is enabled if both "fphp" and "asimdhp" features are supported. + // Only the Neon instructions need this check. SVE supports half-precision floats + // by default. + if (UseSVE == 0 && !is_feat_fp16_supported()) { + return false; + } + break; + case Op_FmaVHF: + // UseFMA flag needs to be checked along with FEAT_FP16 + if (!UseFMA || (UseSVE == 0 && !is_feat_fp16_supported())) { + return false; + } + break; default: break; } @@ -260,6 +280,19 @@ source %{ case Op_StoreVectorScatter: opcode = Op_StoreVectorScatterMasked; break; + // Currently, the masked versions of the following 8 Float16 operations are disabled. + // When the support for Float16 vector classes is added in VectorAPI and the masked + // Float16 IR can be generated, these masked operations will be enabled and relevant + // backend support added. + case Op_AddVHF: + case Op_SubVHF: + case Op_MulVHF: + case Op_DivVHF: + case Op_MaxVHF: + case Op_MinVHF: + case Op_SqrtVHF: + case Op_FmaVHF: + return false; default: break; } @@ -508,12 +541,13 @@ dnl // ------------------------------ Vector add ----------------------------------- // vector add -BINARY_OP(vaddB, AddVB, addv, sve_add, B) -BINARY_OP(vaddS, AddVS, addv, sve_add, H) -BINARY_OP(vaddI, AddVI, addv, sve_add, S) -BINARY_OP(vaddL, AddVL, addv, sve_add, D) -BINARY_OP(vaddF, AddVF, fadd, sve_fadd, S) -BINARY_OP(vaddD, AddVD, fadd, sve_fadd, D) +BINARY_OP(vaddB, AddVB, addv, sve_add, B) +BINARY_OP(vaddS, AddVS, addv, sve_add, H) +BINARY_OP(vaddI, AddVI, addv, sve_add, S) +BINARY_OP(vaddL, AddVL, addv, sve_add, D) +BINARY_OP(vaddHF, AddVHF, fadd, sve_fadd, H) +BINARY_OP(vaddF, AddVF, fadd, sve_fadd, S) +BINARY_OP(vaddD, AddVD, fadd, sve_fadd, D) // vector add - predicated BINARY_OP_PREDICATE(vaddB, AddVB, sve_add, B) @@ -532,12 +566,13 @@ VADD_IMM(L, immLAddSubV, D) // ------------------------------ Vector sub ----------------------------------- // vector sub -BINARY_OP(vsubB, SubVB, subv, sve_sub, B) -BINARY_OP(vsubS, SubVS, subv, sve_sub, H) -BINARY_OP(vsubI, SubVI, subv, sve_sub, S) -BINARY_OP(vsubL, SubVL, subv, sve_sub, D) -BINARY_OP(vsubF, SubVF, fsub, sve_fsub, S) -BINARY_OP(vsubD, SubVD, fsub, sve_fsub, D) +BINARY_OP(vsubB, SubVB, subv, sve_sub, B) +BINARY_OP(vsubS, SubVS, subv, sve_sub, H) +BINARY_OP(vsubI, SubVI, subv, sve_sub, S) +BINARY_OP(vsubL, SubVL, subv, sve_sub, D) +BINARY_OP(vsubHF, SubVHF, fsub, sve_fsub, H) +BINARY_OP(vsubF, SubVF, fsub, sve_fsub, S) +BINARY_OP(vsubD, SubVD, fsub, sve_fsub, D) // vector sub - predicated BINARY_OP_PREDICATE(vsubB, SubVB, sve_sub, B) @@ -612,8 +647,9 @@ instruct vmulL_sve(vReg dst_src1, vReg src2) %{ %} // vector mul - floating-point -BINARY_OP(vmulF, MulVF, fmul, sve_fmul, S) -BINARY_OP(vmulD, MulVD, fmul, sve_fmul, D) +BINARY_OP(vmulHF, MulVHF, fmul, sve_fmul, H) +BINARY_OP(vmulF, MulVF, fmul, sve_fmul, S) +BINARY_OP(vmulD, MulVD, fmul, sve_fmul, D) // vector mul - predicated BINARY_OP_PREDICATE(vmulB, MulVB, sve_mul, B) @@ -626,8 +662,9 @@ BINARY_OP_PREDICATE(vmulD, MulVD, sve_fmul, D) // ------------------------------ Vector float div ----------------------------- // vector float div -BINARY_OP_NEON_SVE_PAIRWISE(vdivF, DivVF, fdiv, sve_fdiv, S) -BINARY_OP_NEON_SVE_PAIRWISE(vdivD, DivVD, fdiv, sve_fdiv, D) +BINARY_OP_NEON_SVE_PAIRWISE(vdivHF, DivVHF, fdiv, sve_fdiv, H) +BINARY_OP_NEON_SVE_PAIRWISE(vdivF, DivVF, fdiv, sve_fdiv, S) +BINARY_OP_NEON_SVE_PAIRWISE(vdivD, DivVD, fdiv, sve_fdiv, D) // vector float div - predicated BINARY_OP_PREDICATE(vdivF, DivVF, sve_fdiv, S) @@ -1016,8 +1053,9 @@ UNARY_OP_PREDICATE_WITH_SIZE(vnegD, NegVD, sve_fneg, D) // ------------------------------ Vector sqrt ---------------------------------- // vector sqrt -UNARY_OP(vsqrtF, SqrtVF, fsqrt, sve_fsqrt, S) -UNARY_OP(vsqrtD, SqrtVD, fsqrt, sve_fsqrt, D) +UNARY_OP(vsqrtHF, SqrtVHF, fsqrt, sve_fsqrt, H) +UNARY_OP(vsqrtF, SqrtVF, fsqrt, sve_fsqrt, S) +UNARY_OP(vsqrtD, SqrtVD, fsqrt, sve_fsqrt, D) // vector sqrt - predicated UNARY_OP_PREDICATE_WITH_SIZE(vsqrtF, SqrtVF, sve_fsqrt, S) @@ -1074,6 +1112,20 @@ instruct v$1_neon(vReg dst, vReg src1, vReg src2) %{ ins_pipe(pipe_slow); %}')dnl dnl +dnl VMINMAX_HF_NEON($1, $2, $3 ) +dnl VMINMAX_HF_NEON(type, op_name, insn_fp) +define(`VMINMAX_HF_NEON', ` +instruct v$1_HF_neon(vReg dst, vReg src1, vReg src2) %{ + predicate(VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst ($2 src1 src2)); + format %{ "v$1_HF_neon $dst, $src1, $src2\t# Half float" %} + ins_encode %{ + __ $3($dst$$FloatRegister, get_arrangement(this), + $src1$$FloatRegister, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%}')dnl +dnl dnl VMINMAX_SVE($1, $2, $3, $4 ) dnl VMINMAX_SVE(type, op_name, insn_fp, insn_integral) define(`VMINMAX_SVE', ` @@ -1097,6 +1149,21 @@ instruct v$1_sve(vReg dst_src1, vReg src2) %{ ins_pipe(pipe_slow); %}')dnl dnl +dnl VMINMAX_HF_SVE($1, $2, $3 ) +dnl VMINMAX_HF_SVE(type, op_name, insn_fp) +define(`VMINMAX_HF_SVE', ` +instruct v$1_HF_sve(vReg dst_src1, vReg src2) %{ + predicate(!VM_Version::use_neon_for_vector(Matcher::vector_length_in_bytes(n))); + match(Set dst_src1 ($2 dst_src1 src2)); + format %{ "v$1_HF_sve $dst_src1, $dst_src1, $src2\t# Half float" %} + ins_encode %{ + assert(UseSVE > 0, "must be sve"); + __ $3($dst_src1$$FloatRegister, __ H, + ptrue, $src2$$FloatRegister); + %} + ins_pipe(pipe_slow); +%}')dnl +dnl dnl VMINMAX_PREDICATE($1, $2, $3, $4 ) dnl VMINMAX_PREDICATE(type, op_name, insn_fp, insn_integral) define(`VMINMAX_PREDICATE', ` @@ -1175,9 +1242,11 @@ dnl VMINMAX_L_NEON(min, MinV) VMINMAX_L_SVE(min, MinV, sve_smin) -// vector min - B/S/I/F/D +// vector min - B/S/I/HF/F/D VMINMAX_NEON(min, MinV, fmin, minv) VMINMAX_SVE(min, MinV, sve_fmin, sve_smin) +VMINMAX_HF_NEON(min, MinVHF, fmin) +VMINMAX_HF_SVE(min, MinVHF, sve_fmin) // vector min - predicated VMINMAX_PREDICATE(min, MinV, sve_fmin, sve_smin) @@ -1199,9 +1268,11 @@ VUMINMAX_PREDICATE(umin, UMinV, sve_umin) VMINMAX_L_NEON(max, MaxV) VMINMAX_L_SVE(max, MaxV, sve_smax) -// vector max - B/S/I/F/D +// vector max - B/S/I/HF/F/D VMINMAX_NEON(max, MaxV, fmax, maxv) VMINMAX_SVE(max, MaxV, sve_fmax, sve_smax) +VMINMAX_HF_NEON(max, MaxVHF, fmax) +VMINMAX_HF_SVE(max, MaxVHF, sve_fmax) // vector max - predicated VMINMAX_PREDICATE(max, MaxV, sve_fmax, sve_smax) @@ -1273,8 +1344,9 @@ instruct vmla_masked(vReg dst_src1, vReg src2, vReg src3, pRegGov pg) %{ // dst_src1 = src2 * src3 + dst_src1 instruct vfmla(vReg dst_src1, vReg src2, vReg src3) %{ - match(Set dst_src1 (FmaVF dst_src1 (Binary src2 src3))); - match(Set dst_src1 (FmaVD dst_src1 (Binary src2 src3))); + match(Set dst_src1 (FmaVHF dst_src1 (Binary src2 src3))); + match(Set dst_src1 (FmaVF dst_src1 (Binary src2 src3))); + match(Set dst_src1 (FmaVD dst_src1 (Binary src2 src3))); format %{ "vfmla $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -2938,6 +3010,23 @@ REPLICATE_INT(L, iRegL) REPLICATE_FP(F, S, T_FLOAT ) REPLICATE_FP(D, D, T_DOUBLE) +// Replicate a half-precision float value held in a floating point register +instruct replicateHF(vReg dst, vRegF src) %{ + predicate(Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst (Replicate src)); + format %{ "replicateHF $dst, $src\t# replicate half-precision float" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ dup($dst$$FloatRegister, get_arrangement(this), $src$$FloatRegister); + } else { // length_in_bytes must be > 16 and SVE should be enabled + assert(UseSVE > 0, "must be sve"); + __ sve_cpy($dst$$FloatRegister, __ H, ptrue, $src$$FloatRegister); + } + %} + ins_pipe(pipe_slow); +%} + // replicate from imm instruct replicateI_imm_le128b(vReg dst, immI con) %{ @@ -3006,6 +3095,23 @@ instruct replicateL_imm8_gt128b(vReg dst, immL8_shift8 con) %{ ins_pipe(pipe_slow); %} +// Replicate a 16-bit half precision float value +instruct replicateHF_imm(vReg dst, immH con) %{ + match(Set dst (Replicate con)); + format %{ "replicateHF_imm $dst, $con\t# replicate immediate half-precision float" %} + ins_encode %{ + uint length_in_bytes = Matcher::vector_length_in_bytes(this); + int imm = (int)($con$$constant) & 0xffff; + if (VM_Version::use_neon_for_vector(length_in_bytes)) { + __ mov($dst$$FloatRegister, get_arrangement(this), imm); + } else { // length_in_bytes must be > 16 and SVE should be enabled + assert(UseSVE > 0, "must be sve"); + __ sve_dup($dst$$FloatRegister, __ H, imm); + } + %} + ins_pipe(pipe_slow); +%} + // ------------------------------ Vector insert -------------------------------- // BYTE, SHORT, INT diff --git a/src/hotspot/cpu/aarch64/assembler_aarch64.cpp b/src/hotspot/cpu/aarch64/assembler_aarch64.cpp index b03344f2d80..5e5d6c16b45 100644 --- a/src/hotspot/cpu/aarch64/assembler_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/assembler_aarch64.cpp @@ -240,6 +240,19 @@ void Assembler::add_sub_immediate(Instruction_aarch64 ¤t_insn, srf(Rn, 5); } +// This method is used to generate Advanced SIMD data processing instructions +void Assembler::adv_simd_three_same(Instruction_aarch64 ¤t_insn, FloatRegister Vd, + SIMD_Arrangement T, FloatRegister Vn, FloatRegister Vm, + int op1, int op2, int op3) { + assert(T == T4H || T == T8H || T == T2S || T == T4S || T == T2D, "invalid arrangement"); + int op22 = (T == T2S || T == T4S) ? 0b0 : 0b1; + int op21 = (T == T4H || T == T8H) ? 0b0 : 0b1; + int op14 = (T == T4H || T == T8H) ? 0b00 : 0b11; + f(0, 31), f((int)T & 1, 30), f(op1, 29), f(0b01110, 28, 24), f(op2, 23); + f(op22, 22); f(op21, 21), rf(Vm, 16), f(op14, 15, 14), f(op3, 13, 10), + rf(Vn, 5), rf(Vd, 0); +} + #undef f #undef sf #undef rf @@ -456,7 +469,7 @@ void Assembler::bang_stack_with_offset(int offset) { Unimplemented(); } bool asm_util::operand_valid_for_immediate_bits(int64_t imm, unsigned nbits) { guarantee(nbits == 8 || nbits == 12, "invalid nbits value"); - uint64_t uimm = (uint64_t)uabs((jlong)imm); + uint64_t uimm = (uint64_t)g_uabs((jlong)imm); if (uimm < (UCONST64(1) << nbits)) return true; if (uimm < (UCONST64(1) << (2 * nbits)) diff --git a/src/hotspot/cpu/aarch64/assembler_aarch64.hpp b/src/hotspot/cpu/aarch64/assembler_aarch64.hpp index 3db7d308844..573b451261f 100644 --- a/src/hotspot/cpu/aarch64/assembler_aarch64.hpp +++ b/src/hotspot/cpu/aarch64/assembler_aarch64.hpp @@ -932,7 +932,7 @@ class Assembler : public AbstractAssembler { static const uint64_t branch_range = NOT_DEBUG(128 * M) DEBUG_ONLY(2 * M); static bool reachable_from_branch_at(address branch, address target) { - return uabs(target - branch) < branch_range; + return g_uabs(target - branch) < branch_range; } // Unconditional branch (immediate) @@ -2032,6 +2032,8 @@ void mvnw(Register Rd, Register Rm, INSN(fsqrtd, 0b01, 0b000011); INSN(fcvtd, 0b01, 0b000100); // Double-precision to single-precision + INSN(fsqrth, 0b11, 0b000011); // Half-precision sqrt + private: void _fcvt_narrow_extend(FloatRegister Vd, SIMD_Arrangement Ta, FloatRegister Vn, SIMD_Arrangement Tb, bool do_extend) { @@ -2059,37 +2061,68 @@ void mvnw(Register Rd, Register Rm, #undef INSN // Floating-point data-processing (2 source) - void data_processing(unsigned op31, unsigned type, unsigned opcode, + void data_processing(unsigned op31, unsigned type, unsigned opcode, unsigned op21, FloatRegister Vd, FloatRegister Vn, FloatRegister Vm) { starti; f(op31, 31, 29); f(0b11110, 28, 24); - f(type, 23, 22), f(1, 21), f(opcode, 15, 10); + f(type, 23, 22), f(op21, 21), f(opcode, 15, 10); rf(Vm, 16), rf(Vn, 5), rf(Vd, 0); } -#define INSN(NAME, op31, type, opcode) \ +#define INSN(NAME, op31, type, opcode, op21) \ void NAME(FloatRegister Vd, FloatRegister Vn, FloatRegister Vm) { \ - data_processing(op31, type, opcode, Vd, Vn, Vm); \ - } - - INSN(fabds, 0b011, 0b10, 0b110101); - INSN(fmuls, 0b000, 0b00, 0b000010); - INSN(fdivs, 0b000, 0b00, 0b000110); - INSN(fadds, 0b000, 0b00, 0b001010); - INSN(fsubs, 0b000, 0b00, 0b001110); - INSN(fmaxs, 0b000, 0b00, 0b010010); - INSN(fmins, 0b000, 0b00, 0b010110); - INSN(fnmuls, 0b000, 0b00, 0b100010); - - INSN(fabdd, 0b011, 0b11, 0b110101); - INSN(fmuld, 0b000, 0b01, 0b000010); - INSN(fdivd, 0b000, 0b01, 0b000110); - INSN(faddd, 0b000, 0b01, 0b001010); - INSN(fsubd, 0b000, 0b01, 0b001110); - INSN(fmaxd, 0b000, 0b01, 0b010010); - INSN(fmind, 0b000, 0b01, 0b010110); - INSN(fnmuld, 0b000, 0b01, 0b100010); + data_processing(op31, type, opcode, op21, Vd, Vn, Vm); \ + } + + INSN(fmuls, 0b000, 0b00, 0b000010, 0b1); + INSN(fdivs, 0b000, 0b00, 0b000110, 0b1); + INSN(fadds, 0b000, 0b00, 0b001010, 0b1); + INSN(fsubs, 0b000, 0b00, 0b001110, 0b1); + INSN(fmaxs, 0b000, 0b00, 0b010010, 0b1); + INSN(fmins, 0b000, 0b00, 0b010110, 0b1); + INSN(fnmuls, 0b000, 0b00, 0b100010, 0b1); + + INSN(fmuld, 0b000, 0b01, 0b000010, 0b1); + INSN(fdivd, 0b000, 0b01, 0b000110, 0b1); + INSN(faddd, 0b000, 0b01, 0b001010, 0b1); + INSN(fsubd, 0b000, 0b01, 0b001110, 0b1); + INSN(fmaxd, 0b000, 0b01, 0b010010, 0b1); + INSN(fmind, 0b000, 0b01, 0b010110, 0b1); + INSN(fnmuld, 0b000, 0b01, 0b100010, 0b1); + + // Half-precision floating-point instructions + INSN(fmulh, 0b000, 0b11, 0b000010, 0b1); + INSN(fdivh, 0b000, 0b11, 0b000110, 0b1); + INSN(faddh, 0b000, 0b11, 0b001010, 0b1); + INSN(fsubh, 0b000, 0b11, 0b001110, 0b1); + INSN(fmaxh, 0b000, 0b11, 0b010010, 0b1); + INSN(fminh, 0b000, 0b11, 0b010110, 0b1); + INSN(fnmulh, 0b000, 0b11, 0b100010, 0b1); +#undef INSN + +// Advanced SIMD scalar three same +#define INSN(NAME, U, size, opcode) \ + void NAME(FloatRegister Vd, FloatRegister Vn, FloatRegister Vm) { \ + starti; \ + f(0b01, 31, 30), f(U, 29), f(0b11110, 28, 24), f(size, 23, 22), f(1, 21); \ + rf(Vm, 16), f(opcode, 15, 11), f(1, 10), rf(Vn, 5), rf(Vd, 0); \ + } + + INSN(fabds, 0b1, 0b10, 0b11010); // Floating-point Absolute Difference (single-precision) + INSN(fabdd, 0b1, 0b11, 0b11010); // Floating-point Absolute Difference (double-precision) + +#undef INSN + +// Advanced SIMD scalar three same FP16 +#define INSN(NAME, U, a, opcode) \ + void NAME(FloatRegister Vd, FloatRegister Vn, FloatRegister Vm) { \ + starti; \ + f(0b01, 31, 30), f(U, 29), f(0b11110, 28, 24), f(a, 23), f(0b10, 22, 21); \ + rf(Vm, 16), f(0b00, 15, 14), f(opcode, 13, 11), f(1, 10), rf(Vn, 5), rf(Vd, 0); \ + } + + INSN(fabdh, 0b1, 0b1, 0b010); // Floating-point Absolute Difference (half-precision float) #undef INSN @@ -2120,6 +2153,7 @@ void mvnw(Register Rd, Register Rm, INSN(fnmaddd, 0b000, 0b01, 1, 0); INSN(fnmsub, 0b000, 0b01, 1, 1); + INSN(fmaddh, 0b000, 0b11, 0, 0); // half-precision fused multiply-add (scalar) #undef INSN // Floating-point conditional select @@ -2709,26 +2743,26 @@ template
#undef INSN -// Advanced SIMD three same -#define INSN(NAME, op1, op2, op3) \ - void NAME(FloatRegister Vd, SIMD_Arrangement T, FloatRegister Vn, FloatRegister Vm) { \ - starti; \ - assert(T == T2S || T == T4S || T == T2D, "invalid arrangement"); \ - f(0, 31), f((int)T & 1, 30), f(op1, 29), f(0b01110, 28, 24), f(op2, 23); \ - f(T==T2D ? 1:0, 22); f(1, 21), rf(Vm, 16), f(op3, 15, 10), rf(Vn, 5), rf(Vd, 0); \ - } - - INSN(fabd, 1, 1, 0b110101); - INSN(fadd, 0, 0, 0b110101); - INSN(fdiv, 1, 0, 0b111111); - INSN(faddp, 1, 0, 0b110101); - INSN(fmul, 1, 0, 0b110111); - INSN(fsub, 0, 1, 0b110101); - INSN(fmla, 0, 0, 0b110011); - INSN(fmls, 0, 1, 0b110011); - INSN(fmax, 0, 0, 0b111101); - INSN(fmin, 0, 1, 0b111101); - INSN(facgt, 1, 1, 0b111011); + // Advanced SIMD three same + void adv_simd_three_same(Instruction_aarch64 ¤t_insn, FloatRegister Vd, + SIMD_Arrangement T, FloatRegister Vn, FloatRegister Vm, + int op1, int op2, int op3); +#define INSN(NAME, op1, op2, op3) \ + void NAME(FloatRegister Vd, SIMD_Arrangement T, FloatRegister Vn, FloatRegister Vm) { \ + starti; \ + adv_simd_three_same(current_insn, Vd, T, Vn, Vm, op1, op2, op3); \ + } + INSN(fabd, 1, 1, 0b0101); + INSN(fadd, 0, 0, 0b0101); + INSN(fdiv, 1, 0, 0b1111); + INSN(faddp, 1, 0, 0b0101); + INSN(fmul, 1, 0, 0b0111); + INSN(fsub, 0, 1, 0b0101); + INSN(fmla, 0, 0, 0b0011); + INSN(fmls, 0, 1, 0b0011); + INSN(fmax, 0, 0, 0b1101); + INSN(fmin, 0, 1, 0b1101); + INSN(facgt, 1, 1, 0b1011); #undef INSN @@ -3228,18 +3262,24 @@ template // parameter "tmask" is a 2-bit mask used to indicate which bits in the size // field are determined by the SIMD_Arrangement. The bit of "tmask" should be // set to 1 if corresponding bit marked as "x" in the ArmARM. -#define INSN(NAME, U, size, tmask, opcode) \ - void NAME(FloatRegister Vd, SIMD_Arrangement T, FloatRegister Vn) { \ - starti; \ - assert((ASSERTION), MSG); \ - f(0, 31), f((int)T & 1, 30), f(U, 29), f(0b01110, 28, 24); \ - f(size | ((int)(T >> 1) & tmask), 23, 22), f(0b10000, 21, 17); \ - f(opcode, 16, 12), f(0b10, 11, 10), rf(Vn, 5), rf(Vd, 0); \ +#define INSN(NAME, U, size, tmask, opcode) \ + void NAME(FloatRegister Vd, SIMD_Arrangement T, FloatRegister Vn) { \ + starti; \ + assert((ASSERTION), MSG); \ + int op22 = (int)(T >> 1) & tmask; \ + int op19 = 0b00; \ + if (tmask == 0b01 && (T == T4H || T == T8H)) { \ + op22 = 0b1; \ + op19 = 0b11; \ + } \ + f(0, 31), f((int)T & 1, 30), f(U, 29), f(0b01110, 28, 24); \ + f(size | op22, 23, 22), f(1, 21), f(op19, 20, 19), f(0b00, 18, 17); \ + f(opcode, 16, 12), f(0b10, 11, 10), rf(Vn, 5), rf(Vd, 0); \ } #define MSG "invalid arrangement" -#define ASSERTION (T == T2S || T == T4S || T == T2D) +#define ASSERTION (T == T4H || T == T8H || T == T2S || T == T4S || T == T2D) INSN(fsqrt, 1, 0b10, 0b01, 0b11111); INSN(fabs, 0, 0b10, 0b01, 0b01111); INSN(fneg, 1, 0b10, 0b01, 0b01111); @@ -3366,7 +3406,7 @@ template #define INSN(NAME, opcode) \ void NAME(FloatRegister Zd, SIMD_RegVariant T, FloatRegister Zn, FloatRegister Zm) { \ starti; \ - assert(T == S || T == D, "invalid register variant"); \ + assert(T == H || T == S || T == D, "invalid register variant"); \ f(0b01100101, 31, 24), f(T, 23, 22), f(0, 21), \ rf(Zm, 16), f(0, 15, 13), f(opcode, 12, 10), rf(Zn, 5), rf(Zd, 0); \ } @@ -3451,7 +3491,7 @@ template // SVE floating-point arithmetic - predicate #define INSN(NAME, op1, op2) \ void NAME(FloatRegister Zd_or_Zdn_or_Vd, SIMD_RegVariant T, PRegister Pg, FloatRegister Zn_or_Zm) { \ - assert(T == S || T == D, "invalid register variant"); \ + assert(T == H || T == S || T == D, "invalid register variant"); \ sve_predicate_reg_insn(op1, op2, Zd_or_Zdn_or_Vd, T, Pg, Zn_or_Zm); \ } diff --git a/src/hotspot/cpu/aarch64/c1_CodeStubs_aarch64.cpp b/src/hotspot/cpu/aarch64/c1_CodeStubs_aarch64.cpp index 2334cbdff24..2e53ecb8058 100644 --- a/src/hotspot/cpu/aarch64/c1_CodeStubs_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/c1_CodeStubs_aarch64.cpp @@ -69,7 +69,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { __ far_call(RuntimeAddress(a)); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); return; } @@ -90,7 +90,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { __ blr(lr); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } PredicateFailedStub::PredicateFailedStub(CodeEmitInfo* info) { @@ -103,7 +103,7 @@ void PredicateFailedStub::emit_code(LIR_Assembler* ce) { __ far_call(RuntimeAddress(a)); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void DivByZeroStub::emit_code(LIR_Assembler* ce) { @@ -274,7 +274,7 @@ void ImplicitNullCheckStub::emit_code(LIR_Assembler* ce) { __ far_call(RuntimeAddress(a)); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } @@ -289,7 +289,7 @@ void SimpleExceptionStub::emit_code(LIR_Assembler* ce) { } __ far_call(RuntimeAddress(Runtime1::entry_for(_stub)), rscratch2); ce->add_call_info_here(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } diff --git a/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp b/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp index 6b1a5a7f1e0..afa2ddb47b4 100644 --- a/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/c1_MacroAssembler_aarch64.cpp @@ -72,16 +72,17 @@ int C1_MacroAssembler::lock_object(Register hdr, Register obj, Register disp_hdr null_check_offset = offset(); - if (DiagnoseSyncOnValueBasedClasses != 0) { - load_klass(hdr, obj); - ldrb(hdr, Address(hdr, Klass::misc_flags_offset())); - tst(hdr, KlassFlags::_misc_is_value_based_class); - br(Assembler::NE, slow_case); - } - if (LockingMode == LM_LIGHTWEIGHT) { lightweight_lock(disp_hdr, obj, hdr, temp, rscratch2, slow_case); } else if (LockingMode == LM_LEGACY) { + + if (DiagnoseSyncOnValueBasedClasses != 0) { + load_klass(hdr, obj); + ldrb(hdr, Address(hdr, Klass::misc_flags_offset())); + tst(hdr, KlassFlags::_misc_is_value_based_class); + br(Assembler::NE, slow_case); + } + Label done; // Load object header ldr(hdr, Address(obj, hdr_offset)); diff --git a/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp b/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp index 605a05a44a7..914967e4009 100644 --- a/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.cpp @@ -107,7 +107,8 @@ address C2_MacroAssembler::arrays_hashcode(Register ary, Register cnt, Register assert(is_power_of_2(unroll_factor), "can't use this value to calculate the jump target PC"); andr(tmp2, cnt, unroll_factor - 1); adr(tmp1, BR_BASE); - sub(tmp1, tmp1, tmp2, ext::sxtw, 3); + // For Cortex-A53 offset is 4 because 2 nops are generated. + sub(tmp1, tmp1, tmp2, ext::sxtw, VM_Version::supports_a53mac() ? 4 : 3); movw(tmp2, 0x1f); br(tmp1); @@ -115,6 +116,11 @@ address C2_MacroAssembler::arrays_hashcode(Register ary, Register cnt, Register for (size_t i = 0; i < unroll_factor; ++i) { load(tmp1, Address(post(ary, type2aelembytes(eltype))), eltype); maddw(result, result, tmp2, tmp1); + // maddw generates an extra nop for Cortex-A53 (see maddw definition in macroAssembler). + // Generate 2nd nop to have 4 instructions per iteration. + if (VM_Version::supports_a53mac()) { + nop(); + } } bind(BR_BASE); subsw(cnt, cnt, unroll_factor); @@ -360,7 +366,7 @@ void C2_MacroAssembler::fast_lock_lightweight(Register obj, Register box, Regist Label slow_path; if (UseObjectMonitorTable) { - // Clear cache in case fast locking succeeds. + // Clear cache in case fast locking succeeds or we need to take the slow-path. str(zr, Address(box, BasicLock::object_monitor_cache_offset_in_bytes())); } @@ -2743,3 +2749,107 @@ bool C2_MacroAssembler::in_scratch_emit_size() { } return MacroAssembler::in_scratch_emit_size(); } + +static void abort_verify_int_in_range(uint idx, jint val, jint lo, jint hi) { + fatal("Invalid CastII, idx: %u, val: %d, lo: %d, hi: %d", idx, val, lo, hi); +} + +void C2_MacroAssembler::verify_int_in_range(uint idx, const TypeInt* t, Register rval, Register rtmp) { + assert(!t->empty() && !t->singleton(), "%s", Type::str(t)); + if (t == TypeInt::INT) { + return; + } + BLOCK_COMMENT("verify_int_in_range {"); + Label L_success, L_failure; + + jint lo = t->_lo; + jint hi = t->_hi; + + if (lo != min_jint && hi != max_jint) { + subsw(rtmp, rval, lo); + br(Assembler::LT, L_failure); + subsw(rtmp, rval, hi); + br(Assembler::LE, L_success); + } else if (lo != min_jint) { + subsw(rtmp, rval, lo); + br(Assembler::GE, L_success); + } else if (hi != max_jint) { + subsw(rtmp, rval, hi); + br(Assembler::LE, L_success); + } else { + ShouldNotReachHere(); + } + + bind(L_failure); + movw(c_rarg0, idx); + mov(c_rarg1, rval); + movw(c_rarg2, lo); + movw(c_rarg3, hi); + reconstruct_frame_pointer(rtmp); + rt_call(CAST_FROM_FN_PTR(address, abort_verify_int_in_range), rtmp); + hlt(0); + + bind(L_success); + BLOCK_COMMENT("} verify_int_in_range"); +} + +static void abort_verify_long_in_range(uint idx, jlong val, jlong lo, jlong hi) { + fatal("Invalid CastLL, idx: %u, val: " JLONG_FORMAT ", lo: " JLONG_FORMAT ", hi: " JLONG_FORMAT, idx, val, lo, hi); +} + +void C2_MacroAssembler::verify_long_in_range(uint idx, const TypeLong* t, Register rval, Register rtmp) { + assert(!t->empty() && !t->singleton(), "%s", Type::str(t)); + if (t == TypeLong::LONG) { + return; + } + BLOCK_COMMENT("verify_long_in_range {"); + Label L_success, L_failure; + + jlong lo = t->_lo; + jlong hi = t->_hi; + + if (lo != min_jlong && hi != max_jlong) { + subs(rtmp, rval, lo); + br(Assembler::LT, L_failure); + subs(rtmp, rval, hi); + br(Assembler::LE, L_success); + } else if (lo != min_jlong) { + subs(rtmp, rval, lo); + br(Assembler::GE, L_success); + } else if (hi != max_jlong) { + subs(rtmp, rval, hi); + br(Assembler::LE, L_success); + } else { + ShouldNotReachHere(); + } + + bind(L_failure); + movw(c_rarg0, idx); + mov(c_rarg1, rval); + mov(c_rarg2, lo); + mov(c_rarg3, hi); + reconstruct_frame_pointer(rtmp); + rt_call(CAST_FROM_FN_PTR(address, abort_verify_long_in_range), rtmp); + hlt(0); + + bind(L_success); + BLOCK_COMMENT("} verify_long_in_range"); +} + +void C2_MacroAssembler::reconstruct_frame_pointer(Register rtmp) { + const int framesize = Compile::current()->output()->frame_size_in_bytes(); + if (PreserveFramePointer) { + // frame pointer is valid +#ifdef ASSERT + // Verify frame pointer value in rfp. + add(rtmp, sp, framesize - 2 * wordSize); + Label L_success; + cmp(rfp, rtmp); + br(Assembler::EQ, L_success); + stop("frame pointer mismatch"); + bind(L_success); +#endif // ASSERT + } else { + add(rfp, sp, framesize - 2 * wordSize); + } +} diff --git a/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.hpp b/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.hpp index e0eaa0b76e6..70e4265c7cc 100644 --- a/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.hpp +++ b/src/hotspot/cpu/aarch64/c2_MacroAssembler_aarch64.hpp @@ -188,4 +188,9 @@ void vector_signum_sve(FloatRegister dst, FloatRegister src, FloatRegister zero, FloatRegister one, FloatRegister vtmp, PRegister pgtmp, SIMD_RegVariant T); + void verify_int_in_range(uint idx, const TypeInt* t, Register val, Register tmp); + void verify_long_in_range(uint idx, const TypeLong* t, Register val, Register tmp); + + void reconstruct_frame_pointer(Register rtmp); + #endif // CPU_AARCH64_C2_MACROASSEMBLER_AARCH64_HPP diff --git a/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp b/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp index 0c2d9a32c8c..3874c8cd54e 100644 --- a/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/compressedKlass_aarch64.cpp @@ -70,7 +70,7 @@ static char* reserve_at_eor_compatible_address(size_t size, bool aslr) { const uint64_t immediate = ((uint64_t)immediates[index]) << 32; assert(immediate > 0 && Assembler::operand_valid_for_logical_immediate(/*is32*/false, immediate), "Invalid immediate %d " UINT64_FORMAT, index, immediate); - result = os::attempt_reserve_memory_at((char*)immediate, size, false); + result = os::attempt_reserve_memory_at((char*)immediate, size, mtNone); if (result == nullptr) { log_trace(metaspace, map)("Failed to attach at " UINT64_FORMAT_X, immediate); } @@ -114,7 +114,7 @@ char* CompressedKlassPointers::reserve_address_space_for_compressed_classes(size if (result == nullptr) { constexpr size_t alignment = nth_bit(32); log_debug(metaspace, map)("Trying to reserve at a 32-bit-aligned address"); - result = os::reserve_memory_aligned(size, alignment, false); + result = os::reserve_memory_aligned(size, alignment, mtNone); } return result; diff --git a/src/hotspot/cpu/aarch64/frame_aarch64.cpp b/src/hotspot/cpu/aarch64/frame_aarch64.cpp index b07fa2fa9df..7ffba17bab3 100644 --- a/src/hotspot/cpu/aarch64/frame_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/frame_aarch64.cpp @@ -828,7 +828,6 @@ void JavaFrameAnchor::make_walkable() { // already walkable? if (walkable()) return; vmassert(last_Java_sp() != nullptr, "not called from Java code?"); - vmassert(last_Java_pc() == nullptr, "already walkable"); _last_Java_pc = (address)_last_Java_sp[-1]; vmassert(walkable(), "something went wrong"); } diff --git a/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp b/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp index d22442db0d7..47ae93a4932 100644 --- a/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp +++ b/src/hotspot/cpu/aarch64/frame_aarch64.inline.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 1997, 2025, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2014, Red Hat Inc. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * @@ -35,6 +35,53 @@ // Inline functions for AArch64 frames: +#if INCLUDE_JFR + +// Static helper routines + +inline address frame::interpreter_bcp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::interpreter_frame_bcp_offset]); +} + +inline address frame::interpreter_return_address(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::return_addr_offset]); +} + +inline intptr_t* frame::interpreter_sender_sp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast _ - // Where: - // is the operation name, e.g. sin - // is optional to indicate float/double - // "f/d" for vector float/double operation - // is the number of elements in the vector - // "2/4" for neon, and "x" for sve - // is the precision level - // "u10/u05" represents 1.0/0.5 ULP error bounds - // We use "u10" for all operations by default - // But for those functions do not have u10 support, we use "u05" instead - // indicates neon/sve - // "sve/advsimd" for sve/neon implementations - // e.g. sinfx_u10sve is the method for computing vector float sin using SVE instructions - // cosd2_u10advsimd is the method for computing 2 elements vector double cos using NEON instructions - // - log_info(library)("Loaded library %s, handle " INTPTR_FORMAT, JNI_LIB_PREFIX "sleef" JNI_LIB_SUFFIX, p2i(libsleef)); - - // Math vector stubs implemented with SVE for scalable vector size. - if (UseSVE > 0) { - for (int op = 0; op < VectorSupport::NUM_VECTOR_OP_MATH; op++) { - int vop = VectorSupport::VECTOR_OP_MATH_START + op; - // Skip "tanh" because there is performance regression - if (vop == VectorSupport::VECTOR_OP_TANH) { - continue; - } - - // The native library does not support u10 level of "hypot". - const char* ulf = (vop == VectorSupport::VECTOR_OP_HYPOT) ? "u05" : "u10"; - - snprintf(ebuf, sizeof(ebuf), "%sfx_%ssve", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_SCALABLE][op] = (address)os::dll_lookup(libsleef, ebuf); - - snprintf(ebuf, sizeof(ebuf), "%sdx_%ssve", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_SCALABLE][op] = (address)os::dll_lookup(libsleef, ebuf); - } - } - - // Math vector stubs implemented with NEON for 64/128 bits vector size. - for (int op = 0; op < VectorSupport::NUM_VECTOR_OP_MATH; op++) { - int vop = VectorSupport::VECTOR_OP_MATH_START + op; - // Skip "tanh" because there is performance regression - if (vop == VectorSupport::VECTOR_OP_TANH) { - continue; - } - - // The native library does not support u10 level of "hypot". - const char* ulf = (vop == VectorSupport::VECTOR_OP_HYPOT) ? "u05" : "u10"; - - snprintf(ebuf, sizeof(ebuf), "%sf4_%sadvsimd", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_64][op] = (address)os::dll_lookup(libsleef, ebuf); - - snprintf(ebuf, sizeof(ebuf), "%sf4_%sadvsimd", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_128][op] = (address)os::dll_lookup(libsleef, ebuf); - - snprintf(ebuf, sizeof(ebuf), "%sd2_%sadvsimd", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_128][op] = (address)os::dll_lookup(libsleef, ebuf); - } - } - // Initialization void generate_initial_stubs() { // Generate initial stubs and initializes the entry points @@ -11332,6 +11739,8 @@ class StubGenerator: public StubCodeGenerator { } #endif + StubRoutines::_unsafe_setmemory = generate_unsafe_setmemory(); + StubRoutines::aarch64::set_completed(); // Inidicate that arraycopy and zero_blocks stubs are generated } @@ -11398,12 +11807,10 @@ class StubGenerator: public StubCodeGenerator { StubRoutines::_montgomerySquare = g.generate_multiply(); } - generate_vector_math_stubs(); - #endif // COMPILER2 if (UseChaCha20Intrinsics) { - StubRoutines::_chacha20Block = generate_chacha20Block_qrpar(); + StubRoutines::_chacha20Block = generate_chacha20Block_blockpar(); } if (UseKyberIntrinsics) { @@ -11465,9 +11872,15 @@ class StubGenerator: public StubCodeGenerator { StubRoutines::_sha512_implCompressMB = generate_sha512_implCompress(StubGenStubId::sha512_implCompressMB_id); } if (UseSHA3Intrinsics) { - StubRoutines::_sha3_implCompress = generate_sha3_implCompress(StubGenStubId::sha3_implCompress_id); + StubRoutines::_double_keccak = generate_double_keccak(); - StubRoutines::_sha3_implCompressMB = generate_sha3_implCompress(StubGenStubId::sha3_implCompressMB_id); + if (UseSIMDForSHA3Intrinsic) { + StubRoutines::_sha3_implCompress = generate_sha3_implCompress(StubGenStubId::sha3_implCompress_id); + StubRoutines::_sha3_implCompressMB = generate_sha3_implCompress(StubGenStubId::sha3_implCompressMB_id); + } else { + StubRoutines::_sha3_implCompress = generate_sha3_implCompress_gpr(StubGenStubId::sha3_implCompress_id); + StubRoutines::_sha3_implCompressMB = generate_sha3_implCompress_gpr(StubGenStubId::sha3_implCompressMB_id); + } } if (UsePoly1305Intrinsics) { diff --git a/src/hotspot/cpu/aarch64/templateInterpreterGenerator_aarch64.cpp b/src/hotspot/cpu/aarch64/templateInterpreterGenerator_aarch64.cpp index 2db3b435abb..710970d1ea2 100644 --- a/src/hotspot/cpu/aarch64/templateInterpreterGenerator_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/templateInterpreterGenerator_aarch64.cpp @@ -865,6 +865,10 @@ void TemplateInterpreterGenerator::lock_method() { // rcpool: cp cache // stack_pointer: previous sp void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { + // Save ConstMethod* in r5_const_method for later use to avoid loading multiple times + Register r5_const_method = r5; + __ ldr(r5_const_method, Address(rmethod, Method::const_offset())); + // initialize fixed part of activation frame if (native_call) { __ sub(esp, sp, 14 * wordSize); @@ -875,8 +879,7 @@ void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { __ stp(zr, zr, Address(sp, 12 * wordSize)); } else { __ sub(esp, sp, 12 * wordSize); - __ ldr(rscratch1, Address(rmethod, Method::const_offset())); // get ConstMethod - __ add(rbcp, rscratch1, in_bytes(ConstMethod::codes_offset())); // get codebase + __ add(rbcp, r5_const_method, in_bytes(ConstMethod::codes_offset())); // get codebase __ mov(rscratch1, frame::interpreter_frame_initial_sp_offset); __ stp(rscratch1, rbcp, Address(__ pre(sp, -12 * wordSize))); } @@ -896,9 +899,10 @@ void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { __ stp(rfp, lr, Address(sp, 10 * wordSize)); __ lea(rfp, Address(sp, 10 * wordSize)); - __ ldr(rcpool, Address(rmethod, Method::const_offset())); - __ ldr(rcpool, Address(rcpool, ConstMethod::constants_offset())); - __ ldr(rcpool, Address(rcpool, ConstantPool::cache_offset())); + // Save ConstantPool* in r11_constants for later use to avoid loading multiple times + Register r11_constants = r11; + __ ldr(r11_constants, Address(r5_const_method, ConstMethod::constants_offset())); + __ ldr(rcpool, Address(r11_constants, ConstantPool::cache_offset())); __ sub(rscratch1, rlocals, rfp); __ lsr(rscratch1, rscratch1, Interpreter::logStackElementSize); // rscratch1 = rlocals - fp(); // Store relativized rlocals, see frame::interpreter_frame_locals(). @@ -908,11 +912,12 @@ void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { // leave last_sp as null __ stp(zr, r19_sender_sp, Address(sp, 8 * wordSize)); - // Get mirror - __ load_mirror(r10, rmethod, r5, rscratch2); + // Get mirror. Resolve ConstantPool* -> InstanceKlass* -> Java mirror. + __ ldr(r10, Address(r11_constants, ConstantPool::pool_holder_offset())); + __ ldr(r10, Address(r10, in_bytes(Klass::java_mirror_offset()))); + __ resolve_oop_handle(r10, rscratch1, rscratch2); if (! native_call) { - __ ldr(rscratch1, Address(rmethod, Method::const_offset())); - __ ldrh(rscratch1, Address(rscratch1, ConstMethod::max_stack_offset())); + __ ldrh(rscratch1, Address(r5_const_method, ConstMethod::max_stack_offset())); __ add(rscratch1, rscratch1, MAX2(3, Method::extra_stack_entries())); __ sub(rscratch1, sp, rscratch1, ext::uxtw, 3); __ andr(rscratch1, rscratch1, -16); @@ -1593,6 +1598,30 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) { __ bind(L); } + #if INCLUDE_JFR + __ enter_jfr_critical_section(); + + // This poll test is to uphold the invariant that a JFR sampled frame + // must not return to its caller without a prior safepoint poll check. + // The earlier poll check in this routine is insufficient for this purpose + // because the thread has transitioned back to Java. + + Label slow_path; + Label fast_path; + __ safepoint_poll(slow_path, true /* at_return */, false /* acquire */, false /* in_nmethod */); + __ br(Assembler::AL, fast_path); + __ bind(slow_path); + __ push(dtos); + __ push(ltos); + __ set_last_Java_frame(esp, rfp, __ pc(), rscratch1); + __ super_call_VM_leaf(CAST_FROM_FN_PTR(address, InterpreterRuntime::at_unwind), rthread); + __ reset_last_Java_frame(true); + __ pop(ltos); + __ pop(dtos); + __ bind(fast_path); + +#endif // INCLUDE_JFR + // jvmti support // Note: This must happen _after_ handling/throwing any exceptions since // the exception handler code notifies the runtime of method exits @@ -1615,6 +1644,8 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) { // remove frame anchor __ leave(); + JFR_ONLY(__ leave_jfr_critical_section();) + // restore sender sp __ mov(sp, esp); @@ -1862,6 +1893,7 @@ void TemplateInterpreterGenerator::generate_throw_exception() { Interpreter::_remove_activation_preserving_args_entry = __ pc(); __ empty_expression_stack(); + __ restore_bcp(); // We could have returned from deoptimizing this frame, so restore rbcp. // Set the popframe_processing bit in pending_popframe_condition // indicating that we are currently handling popframe, so that // call_VMs that may happen later do not trigger new popframe diff --git a/src/hotspot/cpu/aarch64/templateTable_aarch64.cpp b/src/hotspot/cpu/aarch64/templateTable_aarch64.cpp index 2cc9b39983a..fcfe153a9a5 100644 --- a/src/hotspot/cpu/aarch64/templateTable_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/templateTable_aarch64.cpp @@ -1144,6 +1144,7 @@ void TemplateTable::aastore() { // Get the value we will store __ ldr(r0, at_tos()); // Now store using the appropriate barrier + // Clobbers: r10, r11, r3 do_oop_store(_masm, element_address, r0, IS_ARRAY); __ b(done); @@ -1152,6 +1153,7 @@ void TemplateTable::aastore() { __ profile_null_seen(r2); // Store a null + // Clobbers: r10, r11, r3 do_oop_store(_masm, element_address, noreg, IS_ARRAY); // Pop stack arguments @@ -1890,6 +1892,8 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) __ mov(r19, r0); // save the nmethod + JFR_ONLY(__ enter_jfr_critical_section();) + call_VM(noreg, CAST_FROM_FN_PTR(address, SharedRuntime::OSR_migration_begin)); // r0 is OSR buffer, move it to expected parameter location @@ -1901,6 +1905,9 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) Address(rfp, frame::interpreter_frame_sender_sp_offset * wordSize)); // remove frame anchor __ leave(); + + JFR_ONLY(__ leave_jfr_critical_section();) + // Ensure compiled code always sees stack at proper alignment __ andr(sp, esp, -16); @@ -2877,6 +2884,7 @@ void TemplateTable::putfield_or_static(int byte_no, bool is_static, RewriteContr __ pop(atos); if (!is_static) pop_and_check_object(obj); // Store into the field + // Clobbers: r10, r11, r3 do_oop_store(_masm, field, r0, IN_HEAP); if (rc == may_rewrite) { patch_bytecode(Bytecodes::_fast_aputfield, bc, r1, true, byte_no); @@ -3072,12 +3080,12 @@ void TemplateTable::fast_storefield(TosState state) // access constant pool cache __ load_field_entry(r2, r1); - // R1: field offset, R2: field holder, R3: flags - load_resolved_field_entry(r2, r2, noreg, r1, r3); + // R1: field offset, R2: field holder, R5: flags + load_resolved_field_entry(r2, r2, noreg, r1, r5); { Label notVolatile; - __ tbz(r3, ResolvedFieldEntry::is_volatile_shift, notVolatile); + __ tbz(r5, ResolvedFieldEntry::is_volatile_shift, notVolatile); __ membar(MacroAssembler::StoreStore | MacroAssembler::LoadStore); __ bind(notVolatile); } @@ -3093,6 +3101,7 @@ void TemplateTable::fast_storefield(TosState state) // access field switch (bytecode()) { case Bytecodes::_fast_aputfield: + // Clobbers: r10, r11, r3 do_oop_store(_masm, field, r0, IN_HEAP); break; case Bytecodes::_fast_lputfield: @@ -3125,7 +3134,7 @@ void TemplateTable::fast_storefield(TosState state) { Label notVolatile; - __ tbz(r3, ResolvedFieldEntry::is_volatile_shift, notVolatile); + __ tbz(r5, ResolvedFieldEntry::is_volatile_shift, notVolatile); __ membar(MacroAssembler::StoreLoad | MacroAssembler::StoreStore); __ bind(notVolatile); } diff --git a/src/hotspot/cpu/aarch64/vmStructs_aarch64.hpp b/src/hotspot/cpu/aarch64/vmStructs_aarch64.hpp index bf9c965213c..2ec901f6a2e 100644 --- a/src/hotspot/cpu/aarch64/vmStructs_aarch64.hpp +++ b/src/hotspot/cpu/aarch64/vmStructs_aarch64.hpp @@ -35,8 +35,7 @@ static_field(VM_Version, _rop_protection, bool) \ static_field(VM_Version, _pac_mask, uintptr_t) -#define VM_TYPES_CPU(declare_type, declare_toplevel_type, declare_oop_type, declare_integer_type, declare_unsigned_integer_type) \ - declare_toplevel_type(VM_Version) +#define VM_TYPES_CPU(declare_type, declare_toplevel_type, declare_oop_type, declare_integer_type, declare_unsigned_integer_type) #define VM_INT_CONSTANTS_CPU(declare_constant, declare_preprocessor_constant) diff --git a/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp b/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp index c555e393ca5..941cb254532 100644 --- a/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/vm_version_aarch64.cpp @@ -161,6 +161,9 @@ void VM_Version::initialize() { (_model == CPU_MODEL_AMPERE_1A || _model == CPU_MODEL_AMPERE_1B)) { FLAG_SET_DEFAULT(CodeEntryAlignment, 32); } + if (FLAG_IS_DEFAULT(AlwaysMergeDMB)) { + FLAG_SET_DEFAULT(AlwaysMergeDMB, false); + } } // ThunderX @@ -376,7 +379,7 @@ void VM_Version::initialize() { FLAG_SET_DEFAULT(UseSHA3Intrinsics, true); } } - } else if (UseSHA3Intrinsics) { + } else if (UseSHA3Intrinsics && UseSIMDForSHA3Intrinsic) { warning("Intrinsics for SHA3-224, SHA3-256, SHA3-384 and SHA3-512 crypto hash functions not available on this CPU."); FLAG_SET_DEFAULT(UseSHA3Intrinsics, false); } @@ -639,6 +642,7 @@ void VM_Version::initialize() { if (_model2) { os::snprintf_checked(buf + buf_used_len, sizeof(buf) - buf_used_len, "(0x%03x)", _model2); } + size_t features_offset = strnlen(buf, sizeof(buf)); #define ADD_FEATURE_IF_SUPPORTED(id, name, bit) \ do { \ if (VM_Version::supports_##name()) strcat(buf, ", " #name); \ @@ -646,7 +650,11 @@ void VM_Version::initialize() { CPU_FEATURE_FLAGS(ADD_FEATURE_IF_SUPPORTED) #undef ADD_FEATURE_IF_SUPPORTED - _features_string = os::strdup(buf); + _cpu_info_string = os::strdup(buf); + + _features_string = extract_features_string(_cpu_info_string, + strnlen(_cpu_info_string, sizeof(buf)), + features_offset); } #if defined(LINUX) @@ -713,7 +721,7 @@ void VM_Version::initialize_cpu_information(void) { int desc_len = snprintf(_cpu_desc, CPU_DETAILED_DESC_BUF_SIZE, "AArch64 "); get_compatible_board(_cpu_desc + desc_len, CPU_DETAILED_DESC_BUF_SIZE - desc_len); desc_len = (int)strlen(_cpu_desc); - snprintf(_cpu_desc + desc_len, CPU_DETAILED_DESC_BUF_SIZE - desc_len, " %s", _features_string); + snprintf(_cpu_desc + desc_len, CPU_DETAILED_DESC_BUF_SIZE - desc_len, " %s", _cpu_info_string); _initialized = true; } diff --git a/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp b/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp index 04cf9c9c2a0..373f8da5405 100644 --- a/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp +++ b/src/hotspot/cpu/aarch64/vm_version_aarch64.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 1997, 2025, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2014, 2020, Red Hat Inc. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * @@ -125,6 +125,8 @@ enum Ampere_CPU_Model { decl(SHA2, sha256, 6) \ decl(CRC32, crc32, 7) \ decl(LSE, lse, 8) \ + decl(FPHP, fphp, 9) \ + decl(ASIMDHP, asimdhp, 10) \ decl(DCPOP, dcpop, 16) \ decl(SHA3, sha3, 17) \ decl(SHA512, sha512, 21) \ diff --git a/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp b/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp index 11ea02621d7..714904ab3df 100644 --- a/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp +++ b/src/hotspot/cpu/aarch64/vtableStubs_aarch64.cpp @@ -25,6 +25,7 @@ #include "asm/assembler.inline.hpp" #include "asm/macroAssembler.inline.hpp" +#include "code/aotCodeCache.hpp" #include "code/compiledIC.hpp" #include "code/vtableStubs.hpp" #include "interp_masm_aarch64.hpp" @@ -196,7 +197,7 @@ VtableStub* VtableStubs::create_itable_stub(int itable_index) { temp_reg, temp_reg2, itable_index, L_no_such_interface); // Reduce "estimate" such that "padding" does not drop below 8. - const ptrdiff_t estimate = 144; + const ptrdiff_t estimate = AOTCodeCache::is_on_for_dump() ? 148 : 144; const ptrdiff_t codesize = __ pc() - start_pc; slop_delta = (int)(estimate - codesize); slop_bytes += slop_delta; diff --git a/src/hotspot/cpu/arm/arm.ad b/src/hotspot/cpu/arm/arm.ad index f3b97d23ad3..4a0b557968c 100644 --- a/src/hotspot/cpu/arm/arm.ad +++ b/src/hotspot/cpu/arm/arm.ad @@ -1238,11 +1238,11 @@ encode %{ enc_class save_last_PC %{ // preserve mark address mark = __ inst_mark(); - debug_only(int off0 = __ offset()); + DEBUG_ONLY(int off0 = __ offset()); int ret_addr_offset = as_MachCall()->ret_addr_offset(); __ adr(LR, mark + ret_addr_offset); __ str(LR, Address(Rthread, JavaThread::last_Java_pc_offset())); - debug_only(int off1 = __ offset()); + DEBUG_ONLY(int off1 = __ offset()); assert(off1 - off0 == 2 * Assembler::InstructionSize, "correct size prediction"); // restore mark __ set_inst_mark(mark); @@ -1251,11 +1251,11 @@ encode %{ enc_class preserve_SP %{ // preserve mark address mark = __ inst_mark(); - debug_only(int off0 = __ offset()); + DEBUG_ONLY(int off0 = __ offset()); // FP is preserved across all calls, even compiled calls. // Use it to preserve SP in places where the callee might change the SP. __ mov(Rmh_SP_save, SP); - debug_only(int off1 = __ offset()); + DEBUG_ONLY(int off1 = __ offset()); assert(off1 - off0 == 4, "correct size prediction"); // restore mark __ set_inst_mark(mark); diff --git a/src/hotspot/cpu/arm/c1_CodeStubs_arm.cpp b/src/hotspot/cpu/arm/c1_CodeStubs_arm.cpp index bca6c7ca30c..5683bc59d5c 100644 --- a/src/hotspot/cpu/arm/c1_CodeStubs_arm.cpp +++ b/src/hotspot/cpu/arm/c1_CodeStubs_arm.cpp @@ -59,7 +59,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { __ call(Runtime1::entry_for(C1StubId::predicate_failed_trap_id), relocInfo::runtime_call_type); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); return; } // Pass the array index on stack because all registers must be preserved @@ -91,7 +91,7 @@ void PredicateFailedStub::emit_code(LIR_Assembler* ce) { __ call(Runtime1::entry_for(C1StubId::predicate_failed_trap_id), relocInfo::runtime_call_type); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void DivByZeroStub::emit_code(LIR_Assembler* ce) { diff --git a/src/hotspot/cpu/arm/frame_arm.hpp b/src/hotspot/cpu/arm/frame_arm.hpp index dee005b8d75..dec27554a47 100644 --- a/src/hotspot/cpu/arm/frame_arm.hpp +++ b/src/hotspot/cpu/arm/frame_arm.hpp @@ -108,6 +108,9 @@ frame(intptr_t* sp, intptr_t* fp); + frame(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc, CodeBlob* cb, bool allow_cb_null = false); + + void setup(address pc); void init(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc); // accessors for the instance variables diff --git a/src/hotspot/cpu/arm/frame_arm.inline.hpp b/src/hotspot/cpu/arm/frame_arm.inline.hpp index 92a48f22f8c..4be190f0504 100644 --- a/src/hotspot/cpu/arm/frame_arm.inline.hpp +++ b/src/hotspot/cpu/arm/frame_arm.inline.hpp @@ -27,9 +27,58 @@ #include "code/codeCache.hpp" #include "code/vmreg.inline.hpp" +#include "runtime/sharedRuntime.hpp" // Inline functions for ARM frames: +#if INCLUDE_JFR + +// Static helper routines + +inline address frame::interpreter_bcp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::interpreter_frame_bcp_offset]); +} + +inline address frame::interpreter_return_address(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::return_addr_offset]); +} + +inline intptr_t* frame::interpreter_sender_sp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast _float_argument_registers; GrowableArray _float_return_registers; - GrowableArray _integer_additional_volatile_registers; - GrowableArray _float_additional_volatile_registers; - int32_t _stack_alignment_bytes; int32_t _shadow_space_bytes; diff --git a/src/hotspot/cpu/ppc/frame_ppc.hpp b/src/hotspot/cpu/ppc/frame_ppc.hpp index 560615089fe..188015f5cd9 100644 --- a/src/hotspot/cpu/ppc/frame_ppc.hpp +++ b/src/hotspot/cpu/ppc/frame_ppc.hpp @@ -1,6 +1,6 @@ /* - * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2024 SAP SE. All rights reserved. + * Copyright (c) 2000, 2025, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2012, 2025 SAP SE. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -134,56 +134,6 @@ #define _native_abi_reg_args_spill(_component) \ (offset_of(frame::native_abi_reg_args_spill, _component)) - // non-volatile GPRs: - - struct spill_nonvolatiles { - uint64_t r14; - uint64_t r15; //_16 - uint64_t r16; - uint64_t r17; //_16 - uint64_t r18; - uint64_t r19; //_16 - uint64_t r20; - uint64_t r21; //_16 - uint64_t r22; - uint64_t r23; //_16 - uint64_t r24; - uint64_t r25; //_16 - uint64_t r26; - uint64_t r27; //_16 - uint64_t r28; - uint64_t r29; //_16 - uint64_t r30; - uint64_t r31; //_16 - - double f14; - double f15; - double f16; - double f17; - double f18; - double f19; - double f20; - double f21; - double f22; - double f23; - double f24; - double f25; - double f26; - double f27; - double f28; - double f29; - double f30; - double f31; - - // aligned to frame::alignment_in_bytes (16) - }; - - enum { - spill_nonvolatiles_size = sizeof(spill_nonvolatiles) - }; - - #define _spill_nonvolatiles_neg(_component) \ - (int)(-frame::spill_nonvolatiles_size + offset_of(frame::spill_nonvolatiles, _component)) // Frame layout for the Java template interpreter on PPC64. // @@ -230,6 +180,7 @@ // [callee's Java result] // [callee's locals w/o arguments] // [outgoing arguments] + // [non-volatiles] // [ENTRY_FRAME_LOCALS] // ABI for every Java frame, compiled and interpreted @@ -292,7 +243,6 @@ uint64_t result_type; uint64_t arguments_tos_address; //_16 // aligned to frame::alignment_in_bytes (16) - uint64_t r[spill_nonvolatiles_size/sizeof(uint64_t)]; }; enum { @@ -413,7 +363,7 @@ inline frame(intptr_t* sp, intptr_t* fp, address pc); inline frame(intptr_t* sp, address pc, kind knd = kind::nmethod); inline frame(intptr_t* sp, address pc, intptr_t* unextended_sp, intptr_t* fp = nullptr, CodeBlob* cb = nullptr); - inline frame(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc, CodeBlob* cb, const ImmutableOopMap* oop_map); + inline frame(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc, CodeBlob* cb, const ImmutableOopMap* oop_map = nullptr); inline frame(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc, CodeBlob* cb, const ImmutableOopMap* oop_map, bool on_heap); private: diff --git a/src/hotspot/cpu/ppc/frame_ppc.inline.hpp b/src/hotspot/cpu/ppc/frame_ppc.inline.hpp index 19a90367353..bb711f2d053 100644 --- a/src/hotspot/cpu/ppc/frame_ppc.inline.hpp +++ b/src/hotspot/cpu/ppc/frame_ppc.inline.hpp @@ -1,6 +1,6 @@ /* - * Copyright (c) 2000, 2024, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2024 SAP SE. All rights reserved. + * Copyright (c) 2000, 2025, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2012, 2025 SAP SE. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -391,4 +391,43 @@ void frame::update_map_with_saved_link(RegisterMapT* map, intptr_t** link_addr) // Nothing to do. } +#if INCLUDE_JFR + +// Static helper routines +inline intptr_t* frame::sender_sp(intptr_t* fp) { return fp; } + +// Extract common_abi parts. +inline intptr_t* frame::fp(const intptr_t* sp) { + assert(sp != nullptr, "invariant"); + return reinterpret_cast #endif // LINUX -// Default value if probing is not implemented for a certain platform: 128TB -static const size_t DEFAULT_MAX_ADDRESS_BIT = 47; -// Minimum value returned, if probing fails: 64GB +// Default value if probing is not implemented for a certain platform +// Max address bit is restricted by implicit assumptions in the code, for instance +// the bit layout of ZForwardingEntry or Partial array entry (see ZMarkStackEntry) in mark stack +static const size_t DEFAULT_MAX_ADDRESS_BIT = 46; +// Minimum value returned, if probing fail static const size_t MINIMUM_MAX_ADDRESS_BIT = 36; static size_t probe_valid_max_address_bit() { @@ -91,10 +93,15 @@ static size_t probe_valid_max_address_bit() { size_t ZPlatformAddressOffsetBits() { static const size_t valid_max_address_offset_bits = probe_valid_max_address_bit() + 1; const size_t max_address_offset_bits = valid_max_address_offset_bits - 3; +#ifdef ADDRESS_SANITIZER + // The max supported value is 44 because of other internal data structures. + return MIN2(valid_max_address_offset_bits, (size_t)44); +#else const size_t min_address_offset_bits = max_address_offset_bits - 2; const size_t address_offset = ZGlobalsPointers::min_address_offset_request(); const size_t address_offset_bits = log2i_exact(address_offset); return clamp(address_offset_bits, min_address_offset_bits, max_address_offset_bits); +#endif } size_t ZPlatformAddressHeapBaseShift() { diff --git a/src/hotspot/cpu/ppc/globals_ppc.hpp b/src/hotspot/cpu/ppc/globals_ppc.hpp index 7fefc856a47..f944408fe29 100644 --- a/src/hotspot/cpu/ppc/globals_ppc.hpp +++ b/src/hotspot/cpu/ppc/globals_ppc.hpp @@ -116,7 +116,7 @@ define_pd_global(intx, InitArrayShortSize, 9*BytesPerLong); \ /* special instructions */ \ product(bool, SuperwordUseVSX, false, \ - "Use Power8 VSX instructions for superword optimization.") \ + "Use VSX instructions for superword optimization.") \ \ product(bool, UseByteReverseInstructions, false, DIAGNOSTIC, \ "Use byte reverse instructions.") \ diff --git a/src/hotspot/cpu/ppc/interp_masm_ppc.hpp b/src/hotspot/cpu/ppc/interp_masm_ppc.hpp index 99ac037e4b7..d3969427db3 100644 --- a/src/hotspot/cpu/ppc/interp_masm_ppc.hpp +++ b/src/hotspot/cpu/ppc/interp_masm_ppc.hpp @@ -1,6 +1,6 @@ /* - * Copyright (c) 2002, 2024, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2023 SAP SE. All rights reserved. + * Copyright (c) 2002, 2025, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2012, 2025 SAP SE. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -170,7 +170,11 @@ class InterpreterMacroAssembler: public MacroAssembler { void remove_activation(TosState state, bool throw_monitor_exception = true, bool install_monitor_exception = true); - void merge_frames(Register Rtop_frame_sp, Register return_pc, Register Rscratch1, Register Rscratch2); // merge top frames + JFR_ONLY(void enter_jfr_critical_section();) + JFR_ONLY(void leave_jfr_critical_section();) + void load_fp(Register fp); + void remove_top_frame_given_fp(Register fp, Register sender_sp, Register sender_fp, Register return_pc, Register temp); + void merge_frames(Register sender_sp, Register return_pc, Register temp1, Register temp2); // merge top frames void add_monitor_to_stack(bool stack_is_empty, Register Rtemp1, Register Rtemp2); diff --git a/src/hotspot/cpu/ppc/interp_masm_ppc_64.cpp b/src/hotspot/cpu/ppc/interp_masm_ppc_64.cpp index 7a75dfd3de1..29fb54250c2 100644 --- a/src/hotspot/cpu/ppc/interp_masm_ppc_64.cpp +++ b/src/hotspot/cpu/ppc/interp_masm_ppc_64.cpp @@ -308,21 +308,11 @@ void InterpreterMacroAssembler::push_2ptrs(Register first, Register second) { } void InterpreterMacroAssembler::move_l_to_d(Register l, FloatRegister d) { - if (VM_Version::has_mtfprd()) { - mtfprd(d, l); - } else { - std(l, 0, R15_esp); - lfd(d, 0, R15_esp); - } + mtfprd(d, l); } void InterpreterMacroAssembler::move_d_to_l(FloatRegister d, Register l) { - if (VM_Version::has_mtfprd()) { - mffprd(l, d); - } else { - stfd(d, 0, R15_esp); - ld(l, 0, R15_esp); - } + mffprd(l, d); } void InterpreterMacroAssembler::push(TosState state) { @@ -793,19 +783,27 @@ void InterpreterMacroAssembler::unlock_if_synchronized_method(TosState state, } // Support function for remove_activation & Co. -void InterpreterMacroAssembler::merge_frames(Register Rsender_sp, Register return_pc, - Register Rscratch1, Register Rscratch2) { - // Pop interpreter frame. - ld(Rscratch1, 0, R1_SP); // *SP - ld(Rsender_sp, _ijava_state_neg(sender_sp), Rscratch1); // top_frame_sp - ld(Rscratch2, 0, Rscratch1); // **SP - if (return_pc!=noreg) { - ld(return_pc, _abi0(lr), Rscratch1); // LR +void InterpreterMacroAssembler::load_fp(Register fp) { + ld(fp, _abi0(callers_sp), R1_SP); // *SP +} + +void InterpreterMacroAssembler::remove_top_frame_given_fp(Register fp, Register sender_sp, Register sender_fp, + Register return_pc, Register temp) { + assert_different_registers(sender_sp, sender_fp, return_pc, temp); + ld(sender_sp, _ijava_state_neg(sender_sp), fp); + ld(sender_fp, _abi0(callers_sp), fp); // **SP + if (return_pc != noreg) { + ld(return_pc, _abi0(lr), fp); // last usage of fp, register can be reused } + subf(temp, R1_SP, sender_sp); // sender_sp - SP + stdux(sender_fp, R1_SP, temp); // atomically set *(SP = sender_sp) = sender_fp +} - // Merge top frames. - subf(Rscratch1, R1_SP, Rsender_sp); // top_frame_sp - SP - stdux(Rscratch2, R1_SP, Rscratch1); // atomically set *(SP = top_frame_sp) = **SP +void InterpreterMacroAssembler::merge_frames(Register sender_sp, Register return_pc, + Register temp1, Register temp2) { + Register fp = temp1, sender_fp = temp2; + load_fp(fp); + remove_top_frame_given_fp(fp, sender_sp, sender_fp, return_pc, /* temp */ fp); } void InterpreterMacroAssembler::narrow(Register result) { @@ -864,11 +862,16 @@ void InterpreterMacroAssembler::remove_activation(TosState state, bool install_monitor_exception) { BLOCK_COMMENT("remove_activation {"); + unlock_if_synchronized_method(state, throw_monitor_exception, install_monitor_exception); + // The below poll is for the stack watermark barrier. It allows fixing up frames lazily, // that would normally not be safe to use. Such bad returns into unsafe territory of // the stack, will call InterpreterRuntime::at_unwind. - Label slow_path; - Label fast_path; + Label slow_path, fast_path; + Register fp = R22_tmp2; + load_fp(fp); + + JFR_ONLY(enter_jfr_critical_section();) safepoint_poll(slow_path, R11_scratch1, true /* at_return */, false /* in_nmethod */); b(fast_path); bind(slow_path); @@ -880,8 +883,6 @@ void InterpreterMacroAssembler::remove_activation(TosState state, align(32); bind(fast_path); - unlock_if_synchronized_method(state, throw_monitor_exception, install_monitor_exception); - // Save result (push state before jvmti call and pop it afterwards) and notify jvmti. notify_method_exit(false, state, NotifyJVMTI, true); @@ -901,10 +902,11 @@ void InterpreterMacroAssembler::remove_activation(TosState state, // call could have a smaller SP, so that this compare succeeds for an // inner call of the method annotated with ReservedStack. ld_ptr(R0, JavaThread::reserved_stack_activation_offset(), R16_thread); - ld_ptr(R11_scratch1, _abi0(callers_sp), R1_SP); // Load frame pointer. - cmpld(CR0, R11_scratch1, R0); + cmpld(CR0, fp, R0); blt_predict_taken(CR0, no_reserved_zone_enabling); + JFR_ONLY(leave_jfr_critical_section();) + // Enable reserved zone again, throw stack overflow exception. call_VM_leaf(CAST_FROM_FN_PTR(address, SharedRuntime::enable_stack_reserved_zone), R16_thread); call_VM(noreg, CAST_FROM_FN_PTR(address, InterpreterRuntime::throw_delayed_StackOverflowError)); @@ -916,12 +918,26 @@ void InterpreterMacroAssembler::remove_activation(TosState state, verify_oop(R17_tos, state); - merge_frames(/*top_frame_sp*/ R21_sender_SP, /*return_pc*/ R0, R11_scratch1, R12_scratch2); + remove_top_frame_given_fp(fp, R21_sender_SP, R23_tmp3, /*return_pc*/ R0, R11_scratch1); mtlr(R0); pop_cont_fastpath(); + JFR_ONLY(leave_jfr_critical_section();) + BLOCK_COMMENT("} remove_activation"); } +#if INCLUDE_JFR +void InterpreterMacroAssembler::enter_jfr_critical_section() { + li(R0, 1); + stb(R0, in_bytes(SAMPLING_CRITICAL_SECTION_OFFSET_JFR), R16_thread); +} + +void InterpreterMacroAssembler::leave_jfr_critical_section() { + li(R0, 0); + stb(R0, in_bytes(SAMPLING_CRITICAL_SECTION_OFFSET_JFR), R16_thread); +} +#endif // INCLUDE_JFR + // Lock object // // Registers alive @@ -958,17 +974,18 @@ void InterpreterMacroAssembler::lock_object(Register monitor, Register object) { // markWord displaced_header = obj->mark().set_unlocked(); - if (DiagnoseSyncOnValueBasedClasses != 0) { - load_klass(tmp, object); - lbz(tmp, in_bytes(Klass::misc_flags_offset()), tmp); - testbitdi(CR0, R0, tmp, exact_log2(KlassFlags::_misc_is_value_based_class)); - bne(CR0, slow_case); - } - if (LockingMode == LM_LIGHTWEIGHT) { lightweight_lock(monitor, object, header, tmp, slow_case); b(done); } else if (LockingMode == LM_LEGACY) { + + if (DiagnoseSyncOnValueBasedClasses != 0) { + load_klass(tmp, object); + lbz(tmp, in_bytes(Klass::misc_flags_offset()), tmp); + testbitdi(CR0, R0, tmp, exact_log2(KlassFlags::_misc_is_value_based_class)); + bne(CR0, slow_case); + } + // Load markWord from object into header. ld(header, oopDesc::mark_offset_in_bytes(), object); diff --git a/src/hotspot/cpu/ppc/javaFrameAnchor_ppc.hpp b/src/hotspot/cpu/ppc/javaFrameAnchor_ppc.hpp index 8b539bc8101..00a6b4cbf95 100644 --- a/src/hotspot/cpu/ppc/javaFrameAnchor_ppc.hpp +++ b/src/hotspot/cpu/ppc/javaFrameAnchor_ppc.hpp @@ -1,6 +1,6 @@ /* - * Copyright (c) 2002, 2023, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2014 SAP SE. All rights reserved. + * Copyright (c) 2002, 2025, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2012, 2025 SAP SE. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -73,6 +73,8 @@ address last_Java_pc(void) { return _last_Java_pc; } + intptr_t* last_Java_fp() const { return *(intptr_t**)_last_Java_sp; } + void set_last_Java_sp(intptr_t* sp) { OrderAccess::release(); _last_Java_sp = sp; } #endif // CPU_PPC_JAVAFRAMEANCHOR_PPC_HPP diff --git a/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp b/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp index f82395b14fb..857911214c5 100644 --- a/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp +++ b/src/hotspot/cpu/ppc/macroAssembler_ppc.cpp @@ -774,93 +774,82 @@ void MacroAssembler::clobber_carg_stack_slots(Register tmp) { } } -// Uses ordering which corresponds to ABI: -// _savegpr0_14: std r14,-144(r1) -// _savegpr0_15: std r15,-136(r1) -// _savegpr0_16: std r16,-128(r1) -void MacroAssembler::save_nonvolatile_gprs(Register dst, int offset) { - std(R14, offset, dst); offset += 8; - std(R15, offset, dst); offset += 8; - std(R16, offset, dst); offset += 8; - std(R17, offset, dst); offset += 8; - std(R18, offset, dst); offset += 8; - std(R19, offset, dst); offset += 8; - std(R20, offset, dst); offset += 8; - std(R21, offset, dst); offset += 8; - std(R22, offset, dst); offset += 8; - std(R23, offset, dst); offset += 8; - std(R24, offset, dst); offset += 8; - std(R25, offset, dst); offset += 8; - std(R26, offset, dst); offset += 8; - std(R27, offset, dst); offset += 8; - std(R28, offset, dst); offset += 8; - std(R29, offset, dst); offset += 8; - std(R30, offset, dst); offset += 8; - std(R31, offset, dst); offset += 8; - - stfd(F14, offset, dst); offset += 8; - stfd(F15, offset, dst); offset += 8; - stfd(F16, offset, dst); offset += 8; - stfd(F17, offset, dst); offset += 8; - stfd(F18, offset, dst); offset += 8; - stfd(F19, offset, dst); offset += 8; - stfd(F20, offset, dst); offset += 8; - stfd(F21, offset, dst); offset += 8; - stfd(F22, offset, dst); offset += 8; - stfd(F23, offset, dst); offset += 8; - stfd(F24, offset, dst); offset += 8; - stfd(F25, offset, dst); offset += 8; - stfd(F26, offset, dst); offset += 8; - stfd(F27, offset, dst); offset += 8; - stfd(F28, offset, dst); offset += 8; - stfd(F29, offset, dst); offset += 8; - stfd(F30, offset, dst); offset += 8; - stfd(F31, offset, dst); -} - -// Uses ordering which corresponds to ABI: -// _restgpr0_14: ld r14,-144(r1) -// _restgpr0_15: ld r15,-136(r1) -// _restgpr0_16: ld r16,-128(r1) -void MacroAssembler::restore_nonvolatile_gprs(Register src, int offset) { - ld(R14, offset, src); offset += 8; - ld(R15, offset, src); offset += 8; - ld(R16, offset, src); offset += 8; - ld(R17, offset, src); offset += 8; - ld(R18, offset, src); offset += 8; - ld(R19, offset, src); offset += 8; - ld(R20, offset, src); offset += 8; - ld(R21, offset, src); offset += 8; - ld(R22, offset, src); offset += 8; - ld(R23, offset, src); offset += 8; - ld(R24, offset, src); offset += 8; - ld(R25, offset, src); offset += 8; - ld(R26, offset, src); offset += 8; - ld(R27, offset, src); offset += 8; - ld(R28, offset, src); offset += 8; - ld(R29, offset, src); offset += 8; - ld(R30, offset, src); offset += 8; - ld(R31, offset, src); offset += 8; - - // FP registers - lfd(F14, offset, src); offset += 8; - lfd(F15, offset, src); offset += 8; - lfd(F16, offset, src); offset += 8; - lfd(F17, offset, src); offset += 8; - lfd(F18, offset, src); offset += 8; - lfd(F19, offset, src); offset += 8; - lfd(F20, offset, src); offset += 8; - lfd(F21, offset, src); offset += 8; - lfd(F22, offset, src); offset += 8; - lfd(F23, offset, src); offset += 8; - lfd(F24, offset, src); offset += 8; - lfd(F25, offset, src); offset += 8; - lfd(F26, offset, src); offset += 8; - lfd(F27, offset, src); offset += 8; - lfd(F28, offset, src); offset += 8; - lfd(F29, offset, src); offset += 8; - lfd(F30, offset, src); offset += 8; - lfd(F31, offset, src); +void MacroAssembler::save_nonvolatile_registers(Register dst, int offset, bool include_fp_regs, bool include_vector_regs) { + BLOCK_COMMENT("save_nonvolatile_registers {"); + + for (int i = 14; i < 32; i++) { + std(as_Register(i), offset, dst); + offset += 8; + } + + if (include_fp_regs) { + for (int i = 14; i < 32; i++) { + stfd(as_FloatRegister(i), offset, dst); + offset += 8; + } + } + + if (include_vector_regs) { + assert(is_aligned(offset, StackAlignmentInBytes), "should be"); + if (PowerArchitecturePPC64 >= 10) { + for (int i = 20; i < 32; i += 2) { + stxvp(as_VectorRegister(i)->to_vsr(), offset, dst); + offset += 32; + } + } else { + for (int i = 20; i < 32; i++) { + if (PowerArchitecturePPC64 >= 9) { + stxv(as_VectorRegister(i)->to_vsr(), offset, dst); + } else { + Register spill_addr = R0; + addi(spill_addr, dst, offset); + stxvd2x(as_VectorRegister(i)->to_vsr(), spill_addr); + } + offset += 16; + } + } + } + + BLOCK_COMMENT("} save_nonvolatile_registers "); +} + +void MacroAssembler::restore_nonvolatile_registers(Register src, int offset, bool include_fp_regs, bool include_vector_regs) { + BLOCK_COMMENT("restore_nonvolatile_registers {"); + + for (int i = 14; i < 32; i++) { + ld(as_Register(i), offset, src); + offset += 8; + } + + if (include_fp_regs) { + for (int i = 14; i < 32; i++) { + lfd(as_FloatRegister(i), offset, src); + offset += 8; + } + } + + if (include_vector_regs) { + assert(is_aligned(offset, StackAlignmentInBytes), "should be"); + if (PowerArchitecturePPC64 >= 10) { + for (int i = 20; i < 32; i += 2) { + lxvp(as_VectorRegister(i)->to_vsr(), offset, src); + offset += 32; + } + } else { + for (int i = 20; i < 32; i++) { + if (PowerArchitecturePPC64 >= 9) { + lxv(as_VectorRegister(i)->to_vsr(), offset, src); + } else { + Register spill_addr = R0; + addi(spill_addr, src, offset); + lxvd2x(as_VectorRegister(i)->to_vsr(), spill_addr); + } + offset += 16; + } + } + } + + BLOCK_COMMENT("} restore_nonvolatile_registers"); } // For verify_oops. @@ -1029,13 +1018,6 @@ void MacroAssembler::push_frame_reg_args(unsigned int bytes, Register tmp) { push_frame(bytes + frame::native_abi_reg_args_size, tmp); } -// Setup up a new C frame with a spill area for non-volatile GPRs and -// additional space for local variables. -void MacroAssembler::push_frame_reg_args_nonvolatiles(unsigned int bytes, - Register tmp) { - push_frame(bytes + frame::native_abi_reg_args_size + frame::spill_nonvolatiles_size, tmp); -} - // Pop current C frame. void MacroAssembler::pop_frame() { ld(R1_SP, _abi0(callers_sp), R1_SP); @@ -1570,57 +1552,27 @@ void MacroAssembler::atomic_get_and_modify_generic(Register dest_current_value, Register addr_base, Register tmp1, Register tmp2, Register tmp3, bool cmpxchgx_hint, bool is_add, int size) { // Sub-word instructions are available since Power 8. - // For older processors, instruction_type != size holds, and we - // emulate the sub-word instructions by constructing a 4-byte value - // that leaves the other bytes unchanged. - const int instruction_type = VM_Version::has_lqarx() ? size : 4; Label retry; Register shift_amount = noreg, val32 = dest_current_value, modval = is_add ? tmp1 : exchange_value; - if (instruction_type != size) { - assert_different_registers(tmp1, tmp2, tmp3, dest_current_value, exchange_value, addr_base); - modval = tmp1; - shift_amount = tmp2; - val32 = tmp3; - // Need some preparation: Compute shift amount, align address. Note: shorts must be 2 byte aligned. -#ifdef VM_LITTLE_ENDIAN - rldic(shift_amount, addr_base, 3, 64-5); // (dest & 3) * 8; - clrrdi(addr_base, addr_base, 2); -#else - xori(shift_amount, addr_base, (size == 1) ? 3 : 2); - clrrdi(addr_base, addr_base, 2); - rldic(shift_amount, shift_amount, 3, 64-5); // byte: ((3-dest) & 3) * 8; short: ((1-dest/2) & 1) * 16; -#endif - } // atomic emulation loop bind(retry); - switch (instruction_type) { + switch (size) { case 4: lwarx(val32, addr_base, cmpxchgx_hint); break; case 2: lharx(val32, addr_base, cmpxchgx_hint); break; case 1: lbarx(val32, addr_base, cmpxchgx_hint); break; default: ShouldNotReachHere(); } - if (instruction_type != size) { - srw(dest_current_value, val32, shift_amount); - } - if (is_add) { add(modval, dest_current_value, exchange_value); } - if (instruction_type != size) { - // Transform exchange value such that the replacement can be done by one xor instruction. - xorr(modval, dest_current_value, is_add ? modval : exchange_value); - clrldi(modval, modval, (size == 1) ? 56 : 48); - slw(modval, modval, shift_amount); - xorr(modval, val32, modval); - } - switch (instruction_type) { + switch (size) { case 4: stwcx_(modval, addr_base); break; case 2: sthcx_(modval, addr_base); break; case 1: stbcx_(modval, addr_base); break; @@ -1645,51 +1597,22 @@ void MacroAssembler::atomic_get_and_modify_generic(Register dest_current_value, // Only signed types are supported with size < 4. void MacroAssembler::cmpxchg_loop_body(ConditionRegister flag, Register dest_current_value, RegisterOrConstant compare_value, Register exchange_value, - Register addr_base, Register tmp1, Register tmp2, - Label &retry, Label &failed, bool cmpxchgx_hint, int size) { + Register addr_base, Label &retry, Label &failed, bool cmpxchgx_hint, int size) { // Sub-word instructions are available since Power 8. - // For older processors, instruction_type != size holds, and we - // emulate the sub-word instructions by constructing a 4-byte value - // that leaves the other bytes unchanged. - const int instruction_type = VM_Version::has_lqarx() ? size : 4; - Register shift_amount = noreg, val32 = dest_current_value, modval = exchange_value; - if (instruction_type != size) { - assert_different_registers(tmp1, tmp2, dest_current_value, compare_value.register_or_noreg(), exchange_value, addr_base); - shift_amount = tmp1; - val32 = tmp2; - modval = tmp2; - // Need some preparation: Compute shift amount, align address. Note: shorts must be 2 byte aligned. -#ifdef VM_LITTLE_ENDIAN - rldic(shift_amount, addr_base, 3, 64-5); // (dest & 3) * 8; - clrrdi(addr_base, addr_base, 2); -#else - xori(shift_amount, addr_base, (size == 1) ? 3 : 2); - clrrdi(addr_base, addr_base, 2); - rldic(shift_amount, shift_amount, 3, 64-5); // byte: ((3-dest) & 3) * 8; short: ((1-dest/2) & 1) * 16; -#endif - // Transform exchange value such that the replacement can be done by one xor instruction. - xorr(exchange_value, compare_value, exchange_value); - clrldi(exchange_value, exchange_value, (size == 1) ? 56 : 48); - slw(exchange_value, exchange_value, shift_amount); - } - // atomic emulation loop bind(retry); - switch (instruction_type) { + switch (size) { case 4: lwarx(val32, addr_base, cmpxchgx_hint); break; case 2: lharx(val32, addr_base, cmpxchgx_hint); break; case 1: lbarx(val32, addr_base, cmpxchgx_hint); break; default: ShouldNotReachHere(); } - if (instruction_type != size) { - srw(dest_current_value, val32, shift_amount); - } if (size == 1) { extsb(dest_current_value, dest_current_value); } else if (size == 2) { @@ -1705,11 +1628,7 @@ void MacroAssembler::cmpxchg_loop_body(ConditionRegister flag, Register dest_cur // branch to done => (flag == ne), (dest_current_value != compare_value) // fall through => (flag == eq), (dest_current_value == compare_value) - if (instruction_type != size) { - xorr(modval, val32, exchange_value); - } - - switch (instruction_type) { + switch (size) { case 4: stwcx_(modval, addr_base); break; case 2: sthcx_(modval, addr_base); break; case 1: stbcx_(modval, addr_base); break; @@ -1720,8 +1639,7 @@ void MacroAssembler::cmpxchg_loop_body(ConditionRegister flag, Register dest_cur // CmpxchgX sets condition register to cmpX(current, compare). void MacroAssembler::cmpxchg_generic(ConditionRegister flag, Register dest_current_value, RegisterOrConstant compare_value, Register exchange_value, - Register addr_base, Register tmp1, Register tmp2, - int semantics, bool cmpxchgx_hint, Register int_flag_success, + Register addr_base, int semantics, bool cmpxchgx_hint, Register int_flag_success, Label* failed_ext, bool contention_hint, bool weak, int size) { Label retry; Label failed_int; @@ -1732,8 +1650,7 @@ void MacroAssembler::cmpxchg_generic(ConditionRegister flag, Register dest_curre // result register is different from the other ones. bool use_result_reg = (int_flag_success != noreg); bool preset_result_reg = (int_flag_success != dest_current_value && int_flag_success != compare_value.register_or_noreg() && - int_flag_success != exchange_value && int_flag_success != addr_base && - int_flag_success != tmp1 && int_flag_success != tmp2); + int_flag_success != exchange_value && int_flag_success != addr_base); assert(!weak || flag == CR0, "weak only supported with CR0"); assert(int_flag_success == noreg || failed_ext == nullptr, "cannot have both"); assert(size == 1 || size == 2 || size == 4, "unsupported"); @@ -1759,7 +1676,7 @@ void MacroAssembler::cmpxchg_generic(ConditionRegister flag, Register dest_curre release(); } - cmpxchg_loop_body(flag, dest_current_value, compare_value, exchange_value, addr_base, tmp1, tmp2, + cmpxchg_loop_body(flag, dest_current_value, compare_value, exchange_value, addr_base, retry, failed, cmpxchgx_hint, size); if (!weak || use_result_reg || failed_ext) { if (UseStaticBranchPredictionInCompareAndSwapPPC64) { @@ -3000,7 +2917,7 @@ void MacroAssembler::compiler_fast_lock_lightweight_object(ConditionRegister fla Label slow_path; if (UseObjectMonitorTable) { - // Clear cache in case fast locking succeeds. + // Clear cache in case fast locking succeeds or we need to take the slow-path. li(tmp1, 0); std(tmp1, in_bytes(BasicObjectLock::lock_offset()) + BasicLock::object_monitor_cache_offset_in_bytes(), box); } @@ -3360,6 +3277,35 @@ void MacroAssembler::safepoint_poll(Label& slow_path, Register temp, bool at_ret } } +void MacroAssembler::jump_to_polling_page_return_handler_blob(int safepoint_offset, bool fixed_size) { + assert(SharedRuntime::polling_page_return_handler_blob() != nullptr, + "polling page return stub not created yet"); + address stub = SharedRuntime::polling_page_return_handler_blob()->entry_point(); + + // Determine saved exception pc using pc relative address computation. + { + Label next_pc; + bl(next_pc); + bind(next_pc); + } + int current_offset = offset(); + + if (fixed_size) { + // Code size must not depend on offsets. + load_const32(R12, safepoint_offset - current_offset); + mflr(R0); + add(R12, R12, R0); + } else { + mflr(R12); + add_const_optimized(R12, R12, safepoint_offset - current_offset); + } + std(R12, in_bytes(JavaThread::saved_exception_pc_offset()), R16_thread); + + add_const_optimized(R0, R29_TOC, MacroAssembler::offset_to_global_toc(stub)); + mtctr(R0); + bctr(); +} + void MacroAssembler::resolve_jobject(Register value, Register tmp1, Register tmp2, MacroAssembler::PreservationLevel preservation_level) { BarrierSetAssembler* bs = BarrierSet::barrier_set()->barrier_set_assembler(); @@ -3735,7 +3681,6 @@ void MacroAssembler::load_reverse_32(Register dst, Register src) { // Due to register shortage, setting tc3 may overwrite table. With the return offset // at hand, the original table address can be easily reconstructed. int MacroAssembler::crc32_table_columns(Register table, Register tc0, Register tc1, Register tc2, Register tc3) { - assert(!VM_Version::has_vpmsumb(), "Vector version should be used instead!"); // Point to 4 byte folding tables (byte-reversed version for Big Endian) // Layout: See StubRoutines::ppc::generate_crc_constants. @@ -3868,103 +3813,6 @@ void MacroAssembler::update_1word_crc32(Register crc, Register buf, Register tab xorr(crc, t0, t2); // Now crc contains the final checksum value. } -/** - * @param crc register containing existing CRC (32-bit) - * @param buf register pointing to input byte buffer (byte*) - * @param len register containing number of bytes - * @param table register pointing to CRC table - * - * uses R9..R12 as work register. Must be saved/restored by caller! - */ -void MacroAssembler::kernel_crc32_1word(Register crc, Register buf, Register len, Register table, - Register t0, Register t1, Register t2, Register t3, - Register tc0, Register tc1, Register tc2, Register tc3, - bool invertCRC) { - assert_different_registers(crc, buf, len, table); - - Label L_mainLoop, L_tail; - Register tmp = t0; - Register data = t0; - Register tmp2 = t1; - const int mainLoop_stepping = 4; - const int tailLoop_stepping = 1; - const int log_stepping = exact_log2(mainLoop_stepping); - const int mainLoop_alignment = 32; // InputForNewCode > 4 ? InputForNewCode : 32; - const int complexThreshold = 2*mainLoop_stepping; - - // Don't test for len <= 0 here. This pathological case should not occur anyway. - // Optimizing for it by adding a test and a branch seems to be a waste of CPU cycles - // for all well-behaved cases. The situation itself is detected and handled correctly - // within update_byteLoop_crc32. - assert(tailLoop_stepping == 1, "check tailLoop_stepping!"); - - BLOCK_COMMENT("kernel_crc32_1word {"); - - if (invertCRC) { - nand(crc, crc, crc); // 1s complement of crc - } - - // Check for short ( + void load_base(uint8_t Rd, Register Rs, const int32_t offset) { + guarantee(is_simm12(offset), "offset is invalid."); + unsigned insn = 0; + int32_t val = offset & 0xfff; + patch((address)&insn, 6, 0, op_major); + patch((address)&insn, 11, 7, Rd); + patch((address)&insn, 14, 12, width); + patch_reg((address)&insn, 15, Rs); + patch((address)&insn, 31, 20, val); + emit(insn); + } + + template + void load_base(Register Rd, Register Rs, const int32_t offset) { + load_base + void load_base(FloatRegister Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void _lbu(Register Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void _lh(Register Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void _lhu(Register Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void _lw(Register Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void lwu(Register Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void _ld(Register Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void flh(FloatRegister Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void flw(FloatRegister Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } + + void _fld(FloatRegister Rd, Register Rs, const int32_t offset) { + load_base (Rd, Rs, offset); + } #define INSN(NAME, op, funct3) \ void NAME(Register Rs1, Register Rs2, const int64_t offset) { \ @@ -724,30 +781,70 @@ class Assembler : public AbstractAssembler { #undef INSN -#define INSN(NAME, REGISTER, op, funct3) \ - void NAME(REGISTER Rs1, Register Rs2, const int32_t offset) { \ - guarantee(is_simm12(offset), "offset is invalid."); \ - unsigned insn = 0; \ - uint32_t val = offset & 0xfff; \ - uint32_t low = val & 0x1f; \ - uint32_t high = (val >> 5) & 0x7f; \ - patch((address)&insn, 6, 0, op); \ - patch((address)&insn, 14, 12, funct3); \ - patch_reg((address)&insn, 15, Rs2); \ - patch_reg((address)&insn, 20, Rs1); \ - patch((address)&insn, 11, 7, low); \ - patch((address)&insn, 31, 25, high); \ - emit(insn); \ - } \ + private: - INSN(_sb, Register, 0b0100011, 0b000); - INSN(_sh, Register, 0b0100011, 0b001); - INSN(_sw, Register, 0b0100011, 0b010); - INSN(_sd, Register, 0b0100011, 0b011); - INSN(fsw, FloatRegister, 0b0100111, 0b010); - INSN(_fsd, FloatRegister, 0b0100111, 0b011); + enum StoreWidthFunct3 : uint8_t { + STORE_WIDTH_BYTE = 0b000, + STORE_WIDTH_HALFWORD = 0b001, + STORE_WIDTH_WORD = 0b010, + STORE_WIDTH_DOUBLEWORD = 0b011, + // 0b100 to 0b111 are reserved for this opcode + }; -#undef INSN + static constexpr uint8_t OP_STORE_MAJOR = 0b0100011; + static constexpr uint8_t OP_FP_STORE_MAJOR = 0b0100111; + + template + void store_base(uint8_t Rs2, Register Rs1, const int32_t offset) { + guarantee(is_simm12(offset), "offset is invalid."); + unsigned insn = 0; + uint32_t val = offset & 0xfff; + uint32_t low = val & 0x1f; + uint32_t high = (val >> 5) & 0x7f; + patch((address)&insn, 6, 0, op_code); + patch((address)&insn, 11, 7, low); + patch((address)&insn, 14, 12, width); + patch_reg((address)&insn, 15, Rs1); + patch((address)&insn, 24, 20, Rs2); + patch((address)&insn, 31, 25, high); + emit(insn); + } + + template + void store_base(Register Rs2, Register Rs1, const int32_t offset) { + store_base + void store_base(FloatRegister Rs2, Register Rs1, const int32_t offset) { + store_base (Rs2, Rs1, offset); + } + + void _sh(Register Rs2, Register Rs1, const int32_t offset) { + store_base (Rs2, Rs1, offset); + } + + void _sw(Register Rs2, Register Rs1, const int32_t offset) { + store_base (Rs2, Rs1, offset); + } + + void _sd(Register Rs2, Register Rs1, const int32_t offset) { + store_base (Rs2, Rs1, offset); + } + + void fsw(FloatRegister Rs2, Register Rs1, const int32_t offset) { + store_base (Rs2, Rs1, offset); + } + + void _fsd(FloatRegister Rs2, Register Rs1, const int32_t offset) { + store_base (Rs2, Rs1, offset); + } #define INSN(NAME, op, funct3) \ void NAME(Register Rd, const uint32_t csr, Register Rs1) { \ @@ -864,81 +961,239 @@ class Assembler : public AbstractAssembler { #undef INSN -enum Aqrl {relaxed = 0b00, rl = 0b01, aq = 0b10, aqrl = 0b11}; - -#define INSN(NAME, op, funct3, funct7) \ - void NAME(Register Rd, Register Rs1, Register Rs2, Aqrl memory_order = aqrl) { \ - unsigned insn = 0; \ - patch((address)&insn, 6, 0, op); \ - patch((address)&insn, 14, 12, funct3); \ - patch_reg((address)&insn, 7, Rd); \ - patch_reg((address)&insn, 15, Rs1); \ - patch_reg((address)&insn, 20, Rs2); \ - patch((address)&insn, 31, 27, funct7); \ - patch((address)&insn, 26, 25, memory_order); \ - emit(insn); \ - } - - INSN(amoswap_w, 0b0101111, 0b010, 0b00001); - INSN(amoadd_w, 0b0101111, 0b010, 0b00000); - INSN(amoxor_w, 0b0101111, 0b010, 0b00100); - INSN(amoand_w, 0b0101111, 0b010, 0b01100); - INSN(amoor_w, 0b0101111, 0b010, 0b01000); - INSN(amomin_w, 0b0101111, 0b010, 0b10000); - INSN(amomax_w, 0b0101111, 0b010, 0b10100); - INSN(amominu_w, 0b0101111, 0b010, 0b11000); - INSN(amomaxu_w, 0b0101111, 0b010, 0b11100); - INSN(amoswap_d, 0b0101111, 0b011, 0b00001); - INSN(amoadd_d, 0b0101111, 0b011, 0b00000); - INSN(amoxor_d, 0b0101111, 0b011, 0b00100); - INSN(amoand_d, 0b0101111, 0b011, 0b01100); - INSN(amoor_d, 0b0101111, 0b011, 0b01000); - INSN(amomin_d, 0b0101111, 0b011, 0b10000); - INSN(amomax_d , 0b0101111, 0b011, 0b10100); - INSN(amominu_d, 0b0101111, 0b011, 0b11000); - INSN(amomaxu_d, 0b0101111, 0b011, 0b11100); - INSN(amocas_w, 0b0101111, 0b010, 0b00101); - INSN(amocas_d, 0b0101111, 0b011, 0b00101); -#undef INSN - -enum operand_size { int8, int16, int32, uint32, int64 }; - -#define INSN(NAME, op, funct3, funct7) \ - void NAME(Register Rd, Register Rs1, Aqrl memory_order = relaxed) { \ - unsigned insn = 0; \ - uint32_t val = memory_order & 0x3; \ - patch((address)&insn, 6, 0, op); \ - patch((address)&insn, 14, 12, funct3); \ - patch_reg((address)&insn, 7, Rd); \ - patch_reg((address)&insn, 15, Rs1); \ - patch((address)&insn, 25, 20, 0b00000); \ - patch((address)&insn, 31, 27, funct7); \ - patch((address)&insn, 26, 25, val); \ - emit(insn); \ - } - - INSN(lr_w, 0b0101111, 0b010, 0b00010); - INSN(lr_d, 0b0101111, 0b011, 0b00010); - -#undef INSN - -#define INSN(NAME, op, funct3, funct7) \ - void NAME(Register Rd, Register Rs1, Register Rs2, Aqrl memory_order = relaxed) { \ - unsigned insn = 0; \ - uint32_t val = memory_order & 0x3; \ - patch((address)&insn, 6, 0, op); \ - patch((address)&insn, 14, 12, funct3); \ - patch_reg((address)&insn, 7, Rd); \ - patch_reg((address)&insn, 15, Rs2); \ - patch_reg((address)&insn, 20, Rs1); \ - patch((address)&insn, 31, 27, funct7); \ - patch((address)&insn, 26, 25, val); \ - emit(insn); \ + enum Aqrl {relaxed = 0b00, rl = 0b01, aq = 0b10, aqrl = 0b11}; + + private: + + enum AmoWidthFunct3 : uint8_t { + AMO_WIDTH_BYTE = 0b000, // Zabha extension + AMO_WIDTH_HALFWORD = 0b001, // Zabha extension + AMO_WIDTH_WORD = 0b010, + AMO_WIDTH_DOUBLEWORD = 0b011, + AMO_WIDTH_QUADWORD = 0b100, + // 0b101 to 0b111 are reserved + }; + + enum AmoOperationFunct5 : uint8_t { + AMO_ADD = 0b00000, + AMO_SWAP = 0b00001, + AMO_LR = 0b00010, + AMO_SC = 0b00011, + AMO_XOR = 0b00100, + AMO_OR = 0b01000, + AMO_AND = 0b01100, + AMO_MIN = 0b10000, + AMO_MAX = 0b10100, + AMO_MINU = 0b11000, + AMO_MAXU = 0b11100, + AMO_CAS = 0b00101 // Zacas + }; + + static constexpr uint32_t OP_AMO_MAJOR = 0b0101111; + + template + void amo_base(Register Rd, Register Rs1, uint8_t Rs2, Aqrl memory_order = aqrl) { + assert(width > AMO_WIDTH_HALFWORD || UseZabha, "Must be"); + assert(funct5 != AMO_CAS || UseZacas, "Must be"); + unsigned insn = 0; + patch((address)&insn, 6, 0, OP_AMO_MAJOR); + patch_reg((address)&insn, 7, Rd); + patch((address)&insn, 14, 12, width); + patch_reg((address)&insn, 15, Rs1); + patch((address)&insn, 24, 20, Rs2); + patch((address)&insn, 26, 25, memory_order); + patch((address)&insn, 31, 27, funct5); + emit(insn); } - INSN(sc_w, 0b0101111, 0b010, 0b00011); - INSN(sc_d, 0b0101111, 0b011, 0b00011); -#undef INSN + template + void amo_base(Register Rd, Register Rs1, Register Rs2, Aqrl memory_order = aqrl) { + amo_base - void fp_load(FloatRegister Rd, Register Rs, const int32_t offset) { - guarantee(is_uimm3(FpWidth), "Rounding mode is out of validity"); - guarantee(is_simm12(offset), "offset is invalid."); - unsigned insn = 0; - uint32_t val = offset & 0xfff; - patch((address)&insn, 6, 0, OP_LOAD_FP); - patch_reg((address)&insn, 7, Rd); - patch((address)&insn, 14, 12, FpWidth); - patch_reg((address)&insn, 15, Rs); - patch((address)&insn, 31, 20, val); - emit(insn); - } - - public: - - void flh(FloatRegister Rd, Register Rs, const int32_t offset) { fp_load<0b001>(Rd, Rs, offset); } - void flw(FloatRegister Rd, Register Rs, const int32_t offset) { fp_load<0b010>(Rd, Rs, offset); } - void _fld(FloatRegister Rd, Register Rs, const int32_t offset) { fp_load<0b011>(Rd, Rs, offset); } - private: template void fp_fm(FloatRegister Rd, FloatRegister Rs1, FloatRegister Rs2, FloatRegister Rs3, RoundingMode rm) { @@ -1904,8 +2136,14 @@ enum VectorMask { INSN(vand_vv, 0b1010111, 0b000, 0b001001); // Vector Single-Width Integer Add and Subtract - INSN(vsub_vv, 0b1010111, 0b000, 0b000010); INSN(vadd_vv, 0b1010111, 0b000, 0b000000); + INSN(vsub_vv, 0b1010111, 0b000, 0b000010); + + // Vector Saturating Integer Add and Subtract + INSN(vsadd_vv, 0b1010111, 0b000, 0b100001); + INSN(vsaddu_vv, 0b1010111, 0b000, 0b100000); + INSN(vssub_vv, 0b1010111, 0b000, 0b100011); + INSN(vssubu_vv, 0b1010111, 0b000, 0b100010); // Vector Register Gather Instructions INSN(vrgather_vv, 0b1010111, 0b000, 0b001100); @@ -2323,6 +2561,7 @@ enum Nf { } // Vector Bit-manipulation used in Cryptography (Zvbb) Extension + INSN(vandn_vx, 0b1010111, 0b100, 0b000001); INSN(vrol_vx, 0b1010111, 0b100, 0b010101); INSN(vror_vx, 0b1010111, 0b100, 0b010100); @@ -2509,17 +2748,17 @@ enum Nf { // wrappers such as 'add' which do the compressing work through 'c_add' depending on the // the operands of the instruction and availability of the RVC hardware extension. // -// 2. 'CompressibleRegion' and 'IncompressibleRegion' are introduced to mark assembler scopes +// 2. 'CompressibleScope' and 'IncompressibleScope' are introduced to mark assembler scopes // within which instructions are qualified or unqualified to be compressed into their 16-bit // versions. An example: // -// CompressibleRegion cr(_masm); +// CompressibleScope scope(_masm); // __ add(...); // this instruction will be compressed into 'c.add' when possible // { -// IncompressibleRegion ir(_masm); +// IncompressibleScope scope(_masm); // __ add(...); // this instruction will not be compressed // { -// CompressibleRegion cr(_masm); +// CompressibleScope scope(_masm); // __ add(...); // this instruction will be compressed into 'c.add' when possible // } // } @@ -2528,40 +2767,40 @@ enum Nf { // distinguish compressed 16-bit instructions from normal 32-bit ones. private: - bool _in_compressible_region; + bool _in_compressible_scope; public: - bool in_compressible_region() const { return _in_compressible_region; } - void set_in_compressible_region(bool b) { _in_compressible_region = b; } + bool in_compressible_scope() const { return _in_compressible_scope; } + void set_in_compressible_scope(bool b) { _in_compressible_scope = b; } public: - // an abstract compressible region - class AbstractCompressibleRegion : public StackObj { + // An abstract compressible scope + class AbstractCompressibleScope : public StackObj { protected: Assembler *_masm; - bool _saved_in_compressible_region; + bool _saved_in_compressible_scope; protected: - AbstractCompressibleRegion(Assembler *_masm) + AbstractCompressibleScope(Assembler *_masm) : _masm(_masm) - , _saved_in_compressible_region(_masm->in_compressible_region()) {} + , _saved_in_compressible_scope(_masm->in_compressible_scope()) {} }; - // a compressible region - class CompressibleRegion : public AbstractCompressibleRegion { + // A compressible scope + class CompressibleScope : public AbstractCompressibleScope { public: - CompressibleRegion(Assembler *_masm) : AbstractCompressibleRegion(_masm) { - _masm->set_in_compressible_region(true); + CompressibleScope(Assembler *_masm) : AbstractCompressibleScope(_masm) { + _masm->set_in_compressible_scope(true); } - ~CompressibleRegion() { - _masm->set_in_compressible_region(_saved_in_compressible_region); + ~CompressibleScope() { + _masm->set_in_compressible_scope(_saved_in_compressible_scope); } }; - // an incompressible region - class IncompressibleRegion : public AbstractCompressibleRegion { + // An incompressible scope + class IncompressibleScope : public AbstractCompressibleScope { public: - IncompressibleRegion(Assembler *_masm) : AbstractCompressibleRegion(_masm) { - _masm->set_in_compressible_region(false); + IncompressibleScope(Assembler *_masm) : AbstractCompressibleScope(_masm) { + _masm->set_in_compressible_scope(false); } - ~IncompressibleRegion() { - _masm->set_in_compressible_region(_saved_in_compressible_region); + ~IncompressibleScope() { + _masm->set_in_compressible_scope(_saved_in_compressible_scope); } }; @@ -2576,13 +2815,13 @@ enum Nf { template void relocate(RelocationHolder const& rspec, Callback emit_insts, int format = 0) { AbstractAssembler::relocate(rspec, format); - IncompressibleRegion ir(this); // relocations + IncompressibleScope scope(this); // relocations emit_insts(); } template void relocate(relocInfo::relocType rtype, Callback emit_insts, int format = 0) { AbstractAssembler::relocate(rtype, format); - IncompressibleRegion ir(this); // relocations + IncompressibleScope scope(this); // relocations emit_insts(); } @@ -3163,7 +3402,7 @@ enum Nf { public: bool do_compress() const { - return UseRVC && in_compressible_region(); + return UseRVC && in_compressible_scope(); } bool do_compress_zcb(Register reg1 = noreg, Register reg2 = noreg) const { @@ -3178,125 +3417,95 @@ enum Nf { // -------------------------- // Load/store register // -------------------------- -#define INSN(NAME) \ - void NAME(Register Rd, Register Rs, const int32_t offset) { \ - /* lw -> c.lwsp/c.lw */ \ - if (do_compress()) { \ - if (is_c_lwswsp(Rs, Rd, offset, true)) { \ - c_lwsp(Rd, offset); \ - return; \ - } else if (is_c_lwsw(Rs, Rd, offset)) { \ - c_lw(Rd, Rs, offset); \ - return; \ - } \ - } \ - _lw(Rd, Rs, offset); \ + void lw(Register Rd, Register Rs, const int32_t offset) { + /* lw -> c.lwsp/c.lw */ + if (do_compress()) { + if (is_c_lwswsp(Rs, Rd, offset, true)) { + c_lwsp(Rd, offset); + return; + } else if (is_c_lwsw(Rs, Rd, offset)) { + c_lw(Rd, Rs, offset); + return; + } + } + _lw(Rd, Rs, offset); } - INSN(lw); - -#undef INSN - // -------------------------- -#define INSN(NAME) \ - void NAME(Register Rd, Register Rs, const int32_t offset) { \ - /* ld -> c.ldsp/c.ld */ \ - if (do_compress()) { \ - if (is_c_ldsdsp(Rs, Rd, offset, true)) { \ - c_ldsp(Rd, offset); \ - return; \ - } else if (is_c_ldsd(Rs, Rd, offset)) { \ - c_ld(Rd, Rs, offset); \ - return; \ - } \ - } \ - _ld(Rd, Rs, offset); \ + void ld(Register Rd, Register Rs, const int32_t offset) { + /* ld -> c.ldsp/c.ld */ + if (do_compress()) { + if (is_c_ldsdsp(Rs, Rd, offset, true)) { + c_ldsp(Rd, offset); + return; + } else if (is_c_ldsd(Rs, Rd, offset)) { + c_ld(Rd, Rs, offset); + return; + } + } + _ld(Rd, Rs, offset); } - INSN(ld); - -#undef INSN - // -------------------------- -#define INSN(NAME) \ - void NAME(FloatRegister Rd, Register Rs, const int32_t offset) { \ - /* fld -> c.fldsp/c.fld */ \ - if (do_compress()) { \ - if (is_c_fldsdsp(Rs, offset)) { \ - c_fldsp(Rd, offset); \ - return; \ - } else if (is_c_fldsd(Rs, Rd, offset)) { \ - c_fld(Rd, Rs, offset); \ - return; \ - } \ - } \ - _fld(Rd, Rs, offset); \ + void fld(FloatRegister Rd, Register Rs, const int32_t offset) { + /* fld -> c.fldsp/c.fld */ + if (do_compress()) { + if (is_c_fldsdsp(Rs, offset)) { + c_fldsp(Rd, offset); + return; + } else if (is_c_fldsd(Rs, Rd, offset)) { + c_fld(Rd, Rs, offset); + return; + } + } + _fld(Rd, Rs, offset); } - INSN(fld); - -#undef INSN - // -------------------------- -#define INSN(NAME) \ - void NAME(Register Rd, Register Rs, const int32_t offset) { \ - /* sd -> c.sdsp/c.sd */ \ - if (do_compress()) { \ - if (is_c_ldsdsp(Rs, Rd, offset, false)) { \ - c_sdsp(Rd, offset); \ - return; \ - } else if (is_c_ldsd(Rs, Rd, offset)) { \ - c_sd(Rd, Rs, offset); \ - return; \ - } \ - } \ - _sd(Rd, Rs, offset); \ + void sd(Register Rs2, Register Rs1, const int32_t offset) { + /* sd -> c.sdsp/c.sd */ + if (do_compress()) { + if (is_c_ldsdsp(Rs1, Rs2, offset, false)) { + c_sdsp(Rs2, offset); + return; + } else if (is_c_ldsd(Rs1, Rs2, offset)) { + c_sd(Rs2, Rs1, offset); + return; + } + } + _sd(Rs2, Rs1, offset); } - INSN(sd); - -#undef INSN - // -------------------------- -#define INSN(NAME) \ - void NAME(Register Rd, Register Rs, const int32_t offset) { \ - /* sw -> c.swsp/c.sw */ \ - if (do_compress()) { \ - if (is_c_lwswsp(Rs, Rd, offset, false)) { \ - c_swsp(Rd, offset); \ - return; \ - } else if (is_c_lwsw(Rs, Rd, offset)) { \ - c_sw(Rd, Rs, offset); \ - return; \ - } \ - } \ - _sw(Rd, Rs, offset); \ + void sw(Register Rs2, Register Rs1, const int32_t offset) { + /* sw -> c.swsp/c.sw */ + if (do_compress()) { + if (is_c_lwswsp(Rs1, Rs2, offset, false)) { + c_swsp(Rs2, offset); + return; + } else if (is_c_lwsw(Rs1, Rs2, offset)) { + c_sw(Rs2, Rs1, offset); + return; + } + } + _sw(Rs2, Rs1, offset); } - INSN(sw); - -#undef INSN - // -------------------------- -#define INSN(NAME) \ - void NAME(FloatRegister Rd, Register Rs, const int32_t offset) { \ - /* fsd -> c.fsdsp/c.fsd */ \ - if (do_compress()) { \ - if (is_c_fldsdsp(Rs, offset)) { \ - c_fsdsp(Rd, offset); \ - return; \ - } else if (is_c_fldsd(Rs, Rd, offset)) { \ - c_fsd(Rd, Rs, offset); \ - return; \ - } \ - } \ - _fsd(Rd, Rs, offset); \ + void fsd(FloatRegister Rs2, Register Rs1, const int32_t offset) { + /* fsd -> c.fsdsp/c.fsd */ + if (do_compress()) { + if (is_c_fldsdsp(Rs1, offset)) { + c_fsdsp(Rs2, offset); + return; + } else if (is_c_fldsd(Rs1, Rs2, offset)) { + c_fsd(Rs2, Rs1, offset); + return; + } + } + _fsd(Rs2, Rs1, offset); } - INSN(fsd); - -#undef INSN - // -------------------------- // Unconditional branch instructions // -------------------------- @@ -3817,7 +4026,7 @@ enum Nf { static const unsigned long branch_range = 1 * M; static bool reachable_from_branch_at(address branch, address target) { - return uabs(target - branch) < branch_range; + return g_uabs(target - branch) < branch_range; } // Decode the given instruction, checking if it's a 16-bit compressed @@ -3831,7 +4040,7 @@ enum Nf { } } - Assembler(CodeBuffer* code) : AbstractAssembler(code), _in_compressible_region(true) {} + Assembler(CodeBuffer* code) : AbstractAssembler(code), _in_compressible_scope(true) {} }; #endif // CPU_RISCV_ASSEMBLER_RISCV_HPP diff --git a/src/hotspot/cpu/riscv/c1_CodeStubs_riscv.cpp b/src/hotspot/cpu/riscv/c1_CodeStubs_riscv.cpp index d55521823ec..ea299181ca7 100644 --- a/src/hotspot/cpu/riscv/c1_CodeStubs_riscv.cpp +++ b/src/hotspot/cpu/riscv/c1_CodeStubs_riscv.cpp @@ -70,7 +70,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { __ far_call(RuntimeAddress(a)); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); return; } @@ -92,7 +92,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { __ rt_call(Runtime1::entry_for(stub_id), ra); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } PredicateFailedStub::PredicateFailedStub(CodeEmitInfo* info) { @@ -105,7 +105,7 @@ void PredicateFailedStub::emit_code(LIR_Assembler* ce) { __ far_call(RuntimeAddress(a)); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void DivByZeroStub::emit_code(LIR_Assembler* ce) { @@ -258,7 +258,7 @@ void ImplicitNullCheckStub::emit_code(LIR_Assembler* ce) { __ far_call(RuntimeAddress(a)); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void SimpleExceptionStub::emit_code(LIR_Assembler* ce) { @@ -272,7 +272,7 @@ void SimpleExceptionStub::emit_code(LIR_Assembler* ce) { } __ far_call(RuntimeAddress(Runtime1::entry_for(_stub))); ce->add_call_info_here(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void ArrayCopyStub::emit_code(LIR_Assembler* ce) { diff --git a/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp b/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp index 76089e8dd45..32b99f56909 100644 --- a/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp +++ b/src/hotspot/cpu/riscv/c1_MacroAssembler_riscv.cpp @@ -61,16 +61,17 @@ int C1_MacroAssembler::lock_object(Register hdr, Register obj, Register disp_hdr null_check_offset = offset(); - if (DiagnoseSyncOnValueBasedClasses != 0) { - load_klass(hdr, obj); - lbu(hdr, Address(hdr, Klass::misc_flags_offset())); - test_bit(temp, hdr, exact_log2(KlassFlags::_misc_is_value_based_class)); - bnez(temp, slow_case, true /* is_far */); - } - if (LockingMode == LM_LIGHTWEIGHT) { lightweight_lock(disp_hdr, obj, hdr, temp, t1, slow_case); } else if (LockingMode == LM_LEGACY) { + + if (DiagnoseSyncOnValueBasedClasses != 0) { + load_klass(hdr, obj); + lbu(hdr, Address(hdr, Klass::misc_flags_offset())); + test_bit(temp, hdr, exact_log2(KlassFlags::_misc_is_value_based_class)); + bnez(temp, slow_case, /* is_far */ true); + } + Label done; // Load object header ld(hdr, Address(obj, hdr_offset)); @@ -348,7 +349,7 @@ void C1_MacroAssembler::verified_entry(bool breakAtEntry) { // first instruction with a jump. For this action to be legal we // must ensure that this first instruction is a J, JAL or NOP. // Make it a NOP. - IncompressibleRegion ir(this); // keep the nop as 4 bytes for patching. + IncompressibleScope scope(this); // keep the nop as 4 bytes for patching. assert_alignment(pc()); nop(); // 4 bytes } diff --git a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp index 99cbcedb8ff..77b4e26cc92 100644 --- a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp +++ b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.cpp @@ -289,7 +289,7 @@ void C2_MacroAssembler::fast_lock_lightweight(Register obj, Register box, Label slow_path; if (UseObjectMonitorTable) { - // Clear cache in case fast locking succeeds. + // Clear cache in case fast locking succeeds or we need to take the slow-path. sd(zr, Address(box, BasicLock::object_monitor_cache_offset_in_bytes())); } @@ -2156,6 +2156,36 @@ void C2_MacroAssembler::enc_cmove(int cmpFlag, Register op1, Register op2, Regis } } +void C2_MacroAssembler::enc_cmove_cmp_fp(int cmpFlag, FloatRegister op1, FloatRegister op2, Register dst, Register src, bool is_single) { + int op_select = cmpFlag & (~unsigned_branch_mask); + + switch (op_select) { + case BoolTest::eq: + cmov_cmp_fp_eq(op1, op2, dst, src, is_single); + break; + case BoolTest::ne: + cmov_cmp_fp_ne(op1, op2, dst, src, is_single); + break; + case BoolTest::le: + cmov_cmp_fp_le(op1, op2, dst, src, is_single); + break; + case BoolTest::ge: + assert(false, "Should go to BoolTest::le case"); + ShouldNotReachHere(); + break; + case BoolTest::lt: + cmov_cmp_fp_lt(op1, op2, dst, src, is_single); + break; + case BoolTest::gt: + assert(false, "Should go to BoolTest::lt case"); + ShouldNotReachHere(); + break; + default: + assert(false, "unsupported compare condition"); + ShouldNotReachHere(); + } +} + // Set dst to NaN if any NaN input. void C2_MacroAssembler::minmax_fp(FloatRegister dst, FloatRegister src1, FloatRegister src2, FLOAT_TYPE ft, bool is_min) { @@ -3080,7 +3110,9 @@ void C2_MacroAssembler::compare_integral_v(VectorRegister vd, VectorRegister src assert(is_integral_type(bt), "unsupported element type"); assert(vm == Assembler::v0_t ? vd != v0 : true, "should be different registers"); vsetvli_helper(bt, vector_length); - vmclr_m(vd); + if (vm == Assembler::v0_t) { + vmclr_m(vd); + } switch (cond) { case BoolTest::eq: vmseq_vv(vd, src1, src2, vm); break; case BoolTest::ne: vmsne_vv(vd, src1, src2, vm); break; @@ -3103,7 +3135,9 @@ void C2_MacroAssembler::compare_fp_v(VectorRegister vd, VectorRegister src1, Vec assert(is_floating_point_type(bt), "unsupported element type"); assert(vm == Assembler::v0_t ? vd != v0 : true, "should be different registers"); vsetvli_helper(bt, vector_length); - vmclr_m(vd); + if (vm == Assembler::v0_t) { + vmclr_m(vd); + } switch (cond) { case BoolTest::eq: vmfeq_vv(vd, src1, src2, vm); break; case BoolTest::ne: vmfne_vv(vd, src1, src2, vm); break; diff --git a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp index a650174d90f..73fceea3805 100644 --- a/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp +++ b/src/hotspot/cpu/riscv/c2_MacroAssembler_riscv.hpp @@ -129,6 +129,10 @@ Register op1, Register op2, Register dst, Register src); + void enc_cmove_cmp_fp(int cmpFlag, + FloatRegister op1, FloatRegister op2, + Register dst, Register src, bool is_single); + void spill(Register r, bool is64, int offset) { is64 ? sd(r, Address(sp, offset)) : sw(r, Address(sp, offset)); diff --git a/src/hotspot/cpu/riscv/c2_globals_riscv.hpp b/src/hotspot/cpu/riscv/c2_globals_riscv.hpp index de3c1b17c8e..79bdc4917c9 100644 --- a/src/hotspot/cpu/riscv/c2_globals_riscv.hpp +++ b/src/hotspot/cpu/riscv/c2_globals_riscv.hpp @@ -43,7 +43,7 @@ define_pd_global(bool, TieredCompilation, COMPILER1_PRESENT(true) NOT define_pd_global(intx, CompileThreshold, 10000); define_pd_global(intx, OnStackReplacePercentage, 140); -define_pd_global(intx, ConditionalMoveLimit, 0); +define_pd_global(intx, ConditionalMoveLimit, 3); define_pd_global(intx, FreqInlineSize, 325); define_pd_global(intx, MinJumpTableSize, 10); define_pd_global(intx, InteriorEntryAlignment, 16); diff --git a/src/hotspot/cpu/riscv/frame_riscv.cpp b/src/hotspot/cpu/riscv/frame_riscv.cpp index 8ee6d11dcaf..e77375434c2 100644 --- a/src/hotspot/cpu/riscv/frame_riscv.cpp +++ b/src/hotspot/cpu/riscv/frame_riscv.cpp @@ -670,7 +670,6 @@ void JavaFrameAnchor::make_walkable() { // already walkable? if (walkable()) { return; } vmassert(last_Java_sp() != nullptr, "not called from Java code?"); - vmassert(last_Java_pc() == nullptr, "already walkable"); _last_Java_pc = (address)_last_Java_sp[-1]; vmassert(walkable(), "something went wrong"); } diff --git a/src/hotspot/cpu/riscv/frame_riscv.inline.hpp b/src/hotspot/cpu/riscv/frame_riscv.inline.hpp index 2e79c89e7b0..fb31760e20b 100644 --- a/src/hotspot/cpu/riscv/frame_riscv.inline.hpp +++ b/src/hotspot/cpu/riscv/frame_riscv.inline.hpp @@ -35,6 +35,53 @@ // Inline functions for RISCV frames: +#if INCLUDE_JFR + +// Static helper routines + +inline address frame::interpreter_bcp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::interpreter_frame_bcp_offset]); +} + +inline address frame::interpreter_return_address(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::return_addr_offset]); +} + +inline intptr_t* frame::interpreter_sender_sp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast #endif // LINUX -// Default value if probe is not implemented for a certain platform: 128TB -static const size_t DEFAULT_MAX_ADDRESS_BIT = 47; -// Minimum value returned, if probing fails: 64GB +// Default value if probing is not implemented for a certain platform +// Max address bit is restricted by implicit assumptions in the code, for instance +// the bit layout of ZForwardingEntry or Partial array entry (see ZMarkStackEntry) in mark stack +static const size_t DEFAULT_MAX_ADDRESS_BIT = 46; +// Minimum value returned, if probing fail static const size_t MINIMUM_MAX_ADDRESS_BIT = 36; static size_t probe_valid_max_address_bit() { diff --git a/src/hotspot/cpu/riscv/globals_riscv.hpp b/src/hotspot/cpu/riscv/globals_riscv.hpp index 3ef084d30fc..d67e05bbb6d 100644 --- a/src/hotspot/cpu/riscv/globals_riscv.hpp +++ b/src/hotspot/cpu/riscv/globals_riscv.hpp @@ -107,6 +107,7 @@ define_pd_global(intx, InlineSmallCode, 1000); product(bool, UseZfh, false, DIAGNOSTIC, "Use Zfh instructions") \ product(bool, UseZfhmin, false, DIAGNOSTIC, "Use Zfhmin instructions") \ product(bool, UseZacas, false, EXPERIMENTAL, "Use Zacas instructions") \ + product(bool, UseZabha, false, EXPERIMENTAL, "Use UseZabha instructions") \ product(bool, UseZcb, false, EXPERIMENTAL, "Use Zcb instructions") \ product(bool, UseZic64b, false, EXPERIMENTAL, "Use Zic64b instructions") \ product(bool, UseZicbom, false, EXPERIMENTAL, "Use Zicbom instructions") \ diff --git a/src/hotspot/cpu/riscv/interp_masm_riscv.cpp b/src/hotspot/cpu/riscv/interp_masm_riscv.cpp index f1f9414d98a..fae34a9c770 100644 --- a/src/hotspot/cpu/riscv/interp_masm_riscv.cpp +++ b/src/hotspot/cpu/riscv/interp_masm_riscv.cpp @@ -497,9 +497,10 @@ void InterpreterMacroAssembler::dispatch_via(TosState state, address* table) { // remove activation // -// Apply stack watermark barrier. // Unlock the receiver if this is a synchronized method. // Unlock any Java monitors from synchronized blocks. +// Apply stack watermark barrier. +// Notify JVMTI. // Remove the activation from the stack. // // If there are locked Java monitors @@ -509,32 +510,14 @@ void InterpreterMacroAssembler::dispatch_via(TosState state, address* table) { // installs IllegalMonitorStateException // Else // no error processing -void InterpreterMacroAssembler::remove_activation( - TosState state, - bool throw_monitor_exception, - bool install_monitor_exception, - bool notify_jvmdi) { +void InterpreterMacroAssembler::remove_activation(TosState state, + bool throw_monitor_exception, + bool install_monitor_exception, + bool notify_jvmdi) { // Note: Registers x13 may be in use for the // result check if synchronized method Label unlocked, unlock, no_unlock; - // The below poll is for the stack watermark barrier. It allows fixing up frames lazily, - // that would normally not be safe to use. Such bad returns into unsafe territory of - // the stack, will call InterpreterRuntime::at_unwind. - Label slow_path; - Label fast_path; - safepoint_poll(slow_path, true /* at_return */, false /* acquire */, false /* in_nmethod */); - j(fast_path); - - bind(slow_path); - push(state); - set_last_Java_frame(esp, fp, (address)pc(), t0); - super_call_VM_leaf(CAST_FROM_FN_PTR(address, InterpreterRuntime::at_unwind), xthread); - reset_last_Java_frame(true); - pop(state); - - bind(fast_path); - // get the value of _do_not_unlock_if_synchronized into x13 const Address do_not_unlock_if_synchronized(xthread, in_bytes(JavaThread::do_not_unlock_if_synchronized_offset())); @@ -655,10 +638,27 @@ void InterpreterMacroAssembler::remove_activation( bind(no_unlock); - // jvmti support - if (notify_jvmdi) { - notify_method_exit(state, NotifyJVMTI); // preserve TOSCA + JFR_ONLY(enter_jfr_critical_section();) + + // The below poll is for the stack watermark barrier. It allows fixing up frames lazily, + // that would normally not be safe to use. Such bad returns into unsafe territory of + // the stack, will call InterpreterRuntime::at_unwind. + Label slow_path; + Label fast_path; + safepoint_poll(slow_path, true /* at_return */, false /* acquire */, false /* in_nmethod */); + j(fast_path); + + bind(slow_path); + push(state); + set_last_Java_frame(esp, fp, pc(), t0); + super_call_VM_leaf(CAST_FROM_FN_PTR(address, InterpreterRuntime::at_unwind), xthread); + reset_last_Java_frame(true); + pop(state); + bind(fast_path); + // JVMTI support. Make sure the safepoint poll test is issued prior. + if (notify_jvmdi) { + notify_method_exit(state, NotifyJVMTI); // preserve TOSCA } else { notify_method_exit(state, SkipNotifyJVMTI); // preserve TOSCA } @@ -677,9 +677,13 @@ void InterpreterMacroAssembler::remove_activation( subw(t0, t0, StackOverflow::stack_guard_enabled); beqz(t0, no_reserved_zone_enabling); + // look for an overflow into the stack reserved zone, i.e. + // interpreter_frame_sender_sp <= JavaThread::reserved_stack_activation ld(t0, Address(xthread, JavaThread::reserved_stack_activation_offset())); ble(t1, t0, no_reserved_zone_enabling); + JFR_ONLY(leave_jfr_critical_section();) + call_VM_leaf( CAST_FROM_FN_PTR(address, SharedRuntime::enable_stack_reserved_zone), xthread); call_VM(noreg, CAST_FROM_FN_PTR(address, @@ -689,11 +693,14 @@ void InterpreterMacroAssembler::remove_activation( bind(no_reserved_zone_enabling); } + // remove frame anchor + leave(); + + JFR_ONLY(leave_jfr_critical_section();) + // restore sender esp mv(esp, t1); - // remove frame anchor - leave(); // If we're returning to interpreted code we will shortly be // adjusting SP to allow some space for ESP. If we're returning to // compiled code the saved sender SP was saved in sender_sp, so this @@ -701,6 +708,19 @@ void InterpreterMacroAssembler::remove_activation( andi(sp, esp, -16); } +#if INCLUDE_JFR +void InterpreterMacroAssembler::enter_jfr_critical_section() { + const Address sampling_critical_section(xthread, in_bytes(SAMPLING_CRITICAL_SECTION_OFFSET_JFR)); + mv(t0, true); + sb(t0, sampling_critical_section); +} + +void InterpreterMacroAssembler::leave_jfr_critical_section() { + const Address sampling_critical_section(xthread, in_bytes(SAMPLING_CRITICAL_SECTION_OFFSET_JFR)); + sb(zr, sampling_critical_section); +} +#endif // INCLUDE_JFR + // Lock object // // Args: @@ -736,17 +756,18 @@ void InterpreterMacroAssembler::lock_object(Register lock_reg) // Load object pointer into obj_reg c_rarg3 ld(obj_reg, Address(lock_reg, obj_offset)); - if (DiagnoseSyncOnValueBasedClasses != 0) { - load_klass(tmp, obj_reg); - lbu(tmp, Address(tmp, Klass::misc_flags_offset())); - test_bit(tmp, tmp, exact_log2(KlassFlags::_misc_is_value_based_class)); - bnez(tmp, slow_case); - } - if (LockingMode == LM_LIGHTWEIGHT) { lightweight_lock(lock_reg, obj_reg, tmp, tmp2, tmp3, slow_case); j(done); } else if (LockingMode == LM_LEGACY) { + + if (DiagnoseSyncOnValueBasedClasses != 0) { + load_klass(tmp, obj_reg); + lbu(tmp, Address(tmp, Klass::misc_flags_offset())); + test_bit(tmp, tmp, exact_log2(KlassFlags::_misc_is_value_based_class)); + bnez(tmp, slow_case); + } + // Load (object->mark() | 1) into swap_reg ld(t0, Address(obj_reg, oopDesc::mark_offset_in_bytes())); ori(swap_reg, t0, 1); @@ -934,47 +955,29 @@ void InterpreterMacroAssembler::set_mdp_data_at(Register mdp_in, void InterpreterMacroAssembler::increment_mdp_data_at(Register mdp_in, - int constant, - bool decrement) { - increment_mdp_data_at(mdp_in, noreg, constant, decrement); + int constant) { + increment_mdp_data_at(mdp_in, noreg, constant); } void InterpreterMacroAssembler::increment_mdp_data_at(Register mdp_in, - Register reg, - int constant, - bool decrement) { + Register index, + int constant) { assert(ProfileInterpreter, "must be profiling interpreter"); - // %%% this does 64bit counters at best it is wasting space - // at worst it is a rare bug when counters overflow - assert_different_registers(t1, t0, mdp_in, reg); + assert_different_registers(t1, t0, mdp_in, index); Address addr1(mdp_in, constant); Address addr2(t1, 0); Address &addr = addr1; - if (reg != noreg) { + if (index != noreg) { la(t1, addr1); - add(t1, t1, reg); + add(t1, t1, index); addr = addr2; } - if (decrement) { - ld(t0, addr); - subi(t0, t0, DataLayout::counter_increment); - Label L; - bltz(t0, L); // skip store if counter underflow - sd(t0, addr); - bind(L); - } else { - assert(DataLayout::counter_increment == 1, - "flow-free idiom only works with 1"); - ld(t0, addr); - addi(t0, t0, DataLayout::counter_increment); - Label L; - blez(t0, L); // skip store if counter overflow - sd(t0, addr); - bind(L); - } + ld(t0, addr); + addi(t0, t0, DataLayout::counter_increment); + sd(t0, addr); } void InterpreterMacroAssembler::set_mdp_flag_at(Register mdp_in, @@ -1514,7 +1517,7 @@ void InterpreterMacroAssembler::call_VM_leaf_base(address entry_point, int number_of_arguments) { // interpreter specific // - // Note: No need to save/restore rbcp & rlocals pointer since these + // Note: No need to save/restore xbcp & xlocals pointer since these // are callee saved registers and no blocking/ GC can happen // in leaf calls. #ifdef ASSERT diff --git a/src/hotspot/cpu/riscv/interp_masm_riscv.hpp b/src/hotspot/cpu/riscv/interp_masm_riscv.hpp index b94140ea990..891db16b243 100644 --- a/src/hotspot/cpu/riscv/interp_masm_riscv.hpp +++ b/src/hotspot/cpu/riscv/interp_masm_riscv.hpp @@ -233,11 +233,8 @@ class InterpreterMacroAssembler: public MacroAssembler { void verify_method_data_pointer(); void set_mdp_data_at(Register mdp_in, int constant, Register value); - void increment_mdp_data_at(Address data, bool decrement = false); - void increment_mdp_data_at(Register mdp_in, int constant, - bool decrement = false); - void increment_mdp_data_at(Register mdp_in, Register reg, int constant, - bool decrement = false); + void increment_mdp_data_at(Register mdp_in, int constant); + void increment_mdp_data_at(Register mdp_in, Register index, int constant); void increment_mask_and_jump(Address counter_addr, int increment, Address mask, Register tmp1, Register tmp2, @@ -290,6 +287,9 @@ class InterpreterMacroAssembler: public MacroAssembler { void notify_method_entry(); void notify_method_exit(TosState state, NotifyMethodExitMode mode); + JFR_ONLY(void enter_jfr_critical_section();) + JFR_ONLY(void leave_jfr_critical_section();) + virtual void _call_Unimplemented(address call_site) { save_bcp(); set_last_Java_frame(esp, fp, (address) pc(), t0); diff --git a/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp b/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp index b5c311c341d..c755d9ae23d 100644 --- a/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp +++ b/src/hotspot/cpu/riscv/macroAssembler_riscv.cpp @@ -256,7 +256,7 @@ void MacroAssembler::dec_held_monitor_count(Register tmp) { } int MacroAssembler::align(int modulus, int extra_offset) { - CompressibleRegion cr(this); + CompressibleScope scope(this); intptr_t before = offset(); while ((offset() + extra_offset) % modulus != 0) { nop(); } return (int)(offset() - before); @@ -417,7 +417,7 @@ void MacroAssembler::set_last_Java_frame(Register last_java_sp, set_last_Java_frame(last_java_sp, last_java_fp, target(L), tmp); } else { L.add_patch_at(code(), locator()); - IncompressibleRegion ir(this); // the label address will be patched back. + IncompressibleScope scope(this); // the label address will be patched back. set_last_Java_frame(last_java_sp, last_java_fp, pc() /* Patched later */, tmp); } } @@ -564,7 +564,7 @@ void MacroAssembler::_verify_oop(Register reg, const char* s, const char* file, // The length of the instruction sequence emitted should not depend // on the address of the char buffer so that the size of mach nodes for // scratch emit and normal emit matches. - IncompressibleRegion ir(this); // Fixed length + IncompressibleScope scope(this); // Fixed length movptr(t0, (address) b); } @@ -604,7 +604,7 @@ void MacroAssembler::_verify_oop_addr(Address addr, const char* s, const char* f // The length of the instruction sequence emitted should not depend // on the address of the char buffer so that the size of mach nodes for // scratch emit and normal emit matches. - IncompressibleRegion ir(this); // Fixed length + IncompressibleScope scope(this); // Fixed length movptr(t0, (address) b); } @@ -775,7 +775,7 @@ void MacroAssembler::unimplemented(const char* what) { } void MacroAssembler::emit_static_call_stub() { - IncompressibleRegion ir(this); // Fixed length: see CompiledDirectCall::to_interp_stub_size(). + IncompressibleScope scope(this); // Fixed length: see CompiledDirectCall::to_interp_stub_size(). // CompiledDirectCall::set_to_interpreted knows the // exact layout of this stub. @@ -907,7 +907,7 @@ void MacroAssembler::la(Register Rd, const Address &adr) { } void MacroAssembler::la(Register Rd, Label &label) { - IncompressibleRegion ir(this); // the label address may be patched back. + IncompressibleScope scope(this); // the label address may be patched back. wrap_label(Rd, label, &MacroAssembler::la); } @@ -971,7 +971,7 @@ void MacroAssembler::j(const address dest, Register temp) { int64_t distance = dest - pc(); // We can't patch C, i.e. if Label wasn't bound we need to patch this jump. - IncompressibleRegion ir(this); + IncompressibleScope scope(this); if (is_simm21(distance) && ((distance % 2) == 0)) { Assembler::jal(x0, distance); } else { @@ -1267,6 +1267,130 @@ void MacroAssembler::cmov_gtu(Register cmp1, Register cmp2, Register dst, Regist bind(no_set); } +// ----------- cmove, compare float ----------- + +// Move src to dst only if cmp1 == cmp2, +// otherwise leave dst unchanged, including the case where one of them is NaN. +// Clarification: +// java code : cmp1 != cmp2 ? dst : src +// transformed to : CMove dst, (cmp1 eq cmp2), dst, src +void MacroAssembler::cmov_cmp_fp_eq(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single) { + if (UseZicond) { + if (is_single) { + feq_s(t0, cmp1, cmp2); + } else { + feq_d(t0, cmp1, cmp2); + } + czero_nez(dst, dst, t0); + czero_eqz(t0 , src, t0); + orr(dst, dst, t0); + return; + } + Label no_set; + if (is_single) { + // jump if cmp1 != cmp2, including the case of NaN + // not jump (i.e. move src to dst) if cmp1 == cmp2 + float_bne(cmp1, cmp2, no_set); + } else { + double_bne(cmp1, cmp2, no_set); + } + mv(dst, src); + bind(no_set); +} + +// Keep dst unchanged only if cmp1 == cmp2, +// otherwise move src to dst, including the case where one of them is NaN. +// Clarification: +// java code : cmp1 == cmp2 ? dst : src +// transformed to : CMove dst, (cmp1 ne cmp2), dst, src +void MacroAssembler::cmov_cmp_fp_ne(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single) { + if (UseZicond) { + if (is_single) { + feq_s(t0, cmp1, cmp2); + } else { + feq_d(t0, cmp1, cmp2); + } + czero_eqz(dst, dst, t0); + czero_nez(t0 , src, t0); + orr(dst, dst, t0); + return; + } + Label no_set; + if (is_single) { + // jump if cmp1 == cmp2 + // not jump (i.e. move src to dst) if cmp1 != cmp2, including the case of NaN + float_beq(cmp1, cmp2, no_set); + } else { + double_beq(cmp1, cmp2, no_set); + } + mv(dst, src); + bind(no_set); +} + +// When cmp1 <= cmp2 or any of them is NaN then dst = src, otherwise, dst = dst +// Clarification +// scenario 1: +// java code : cmp2 < cmp1 ? dst : src +// transformed to : CMove dst, (cmp1 le cmp2), dst, src +// scenario 2: +// java code : cmp1 > cmp2 ? dst : src +// transformed to : CMove dst, (cmp1 le cmp2), dst, src +void MacroAssembler::cmov_cmp_fp_le(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single) { + if (UseZicond) { + if (is_single) { + flt_s(t0, cmp2, cmp1); + } else { + flt_d(t0, cmp2, cmp1); + } + czero_eqz(dst, dst, t0); + czero_nez(t0 , src, t0); + orr(dst, dst, t0); + return; + } + Label no_set; + if (is_single) { + // jump if cmp1 > cmp2 + // not jump (i.e. move src to dst) if cmp1 <= cmp2 or either is NaN + float_bgt(cmp1, cmp2, no_set); + } else { + double_bgt(cmp1, cmp2, no_set); + } + mv(dst, src); + bind(no_set); +} + +// When cmp1 < cmp2 or any of them is NaN then dst = src, otherwise, dst = dst +// Clarification +// scenario 1: +// java code : cmp2 <= cmp1 ? dst : src +// transformed to : CMove dst, (cmp1 lt cmp2), dst, src +// scenario 2: +// java code : cmp1 >= cmp2 ? dst : src +// transformed to : CMove dst, (cmp1 lt cmp2), dst, src +void MacroAssembler::cmov_cmp_fp_lt(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single) { + if (UseZicond) { + if (is_single) { + fle_s(t0, cmp2, cmp1); + } else { + fle_d(t0, cmp2, cmp1); + } + czero_eqz(dst, dst, t0); + czero_nez(t0 , src, t0); + orr(dst, dst, t0); + return; + } + Label no_set; + if (is_single) { + // jump if cmp1 >= cmp2 + // not jump (i.e. move src to dst) if cmp1 < cmp2 or either is NaN + float_bge(cmp1, cmp2, no_set); + } else { + double_bge(cmp1, cmp2, no_set); + } + mv(dst, src); + bind(no_set); +} + // Float compare branch instructions #define INSN(NAME, FLOATCMP, BRANCH) \ @@ -1682,7 +1806,7 @@ void MacroAssembler::vector_update_crc32(Register crc, Register buf, Register le for (int i = 0; i < N; i++) { vmv_x_s(tmp2, vcrc); // in vmv_x_s, the value is sign-extended to SEW bits, but we need zero-extended here. - zext_w(tmp2, tmp2); + zext(tmp2, tmp2, 32); vslidedown_vi(vcrc, vcrc, 1); xorr(crc, crc, tmp2); for (int j = 0; j < W; j++) { @@ -3615,16 +3739,16 @@ void MacroAssembler::check_klass_subtype(Register sub_klass, bind(L_failure); } -void MacroAssembler::safepoint_poll(Label& slow_path, bool at_return, bool acquire, bool in_nmethod) { - ld(t0, Address(xthread, JavaThread::polling_word_offset())); +void MacroAssembler::safepoint_poll(Label& slow_path, bool at_return, bool acquire, bool in_nmethod, Register tmp_reg) { + ld(tmp_reg, Address(xthread, JavaThread::polling_word_offset())); if (acquire) { membar(MacroAssembler::LoadLoad | MacroAssembler::LoadStore); } if (at_return) { - bgtu(in_nmethod ? sp : fp, t0, slow_path, /* is_far */ true); + bgtu(in_nmethod ? sp : fp, tmp_reg, slow_path, /* is_far */ true); } else { - test_bit(t0, t0, exact_log2(SafepointMechanism::poll_bit())); - bnez(t0, slow_path, true /* is_far */); + test_bit(tmp_reg, tmp_reg, exact_log2(SafepointMechanism::poll_bit())); + bnez(tmp_reg, slow_path, /* is_far */ true); } } @@ -3674,7 +3798,7 @@ void MacroAssembler::cmpxchg_obj_header(Register oldv, Register newv, Register o void MacroAssembler::load_reserved(Register dst, Register addr, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire) { switch (size) { case int64: @@ -3695,15 +3819,15 @@ void MacroAssembler::load_reserved(Register dst, void MacroAssembler::store_conditional(Register dst, Register new_val, Register addr, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl release) { switch (size) { case int64: - sc_d(dst, new_val, addr, release); + sc_d(dst, addr, new_val, release); break; case int32: case uint32: - sc_w(dst, new_val, addr, release); + sc_w(dst, addr, new_val, release); break; default: ShouldNotReachHere(); @@ -3712,7 +3836,7 @@ void MacroAssembler::store_conditional(Register dst, void MacroAssembler::cmpxchg_narrow_value_helper(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Register shift, Register mask, Register aligned_addr) { assert(size == int8 || size == int16, "unsupported operand size"); @@ -3742,10 +3866,11 @@ void MacroAssembler::cmpxchg_narrow_value_helper(Register addr, Register expecte // which are forced to work with 4-byte aligned address. void MacroAssembler::cmpxchg_narrow_value(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result, bool result_as_bool, Register tmp1, Register tmp2, Register tmp3) { + assert(!(UseZacas && UseZabha), "Use amocas"); assert_different_registers(addr, expected, new_val, result, tmp1, tmp2, tmp3, t0, t1); Register scratch0 = t0, aligned_addr = t1; @@ -3778,13 +3903,13 @@ void MacroAssembler::cmpxchg_narrow_value(Register addr, Register expected, notr(scratch1, mask); bind(retry); - lr_w(result, aligned_addr, acquire); + load_reserved(result, aligned_addr, operand_size::int32, acquire); andr(scratch0, result, mask); bne(scratch0, expected, fail); andr(scratch0, result, scratch1); // scratch1 is ~mask orr(scratch0, scratch0, new_val); - sc_w(scratch0, scratch0, aligned_addr, release); + store_conditional(scratch0, scratch0, aligned_addr, operand_size::int32, release); bnez(scratch0, retry); } @@ -3816,10 +3941,11 @@ void MacroAssembler::cmpxchg_narrow_value(Register addr, Register expected, // failed. void MacroAssembler::weak_cmpxchg_narrow_value(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result, Register tmp1, Register tmp2, Register tmp3) { + assert(!(UseZacas && UseZabha), "Use amocas"); assert_different_registers(addr, expected, new_val, result, tmp1, tmp2, tmp3, t0, t1); Register scratch0 = t0, aligned_addr = t1; @@ -3850,13 +3976,13 @@ void MacroAssembler::weak_cmpxchg_narrow_value(Register addr, Register expected, } else { notr(scratch1, mask); - lr_w(result, aligned_addr, acquire); + load_reserved(result, aligned_addr, operand_size::int32, acquire); andr(scratch0, result, mask); bne(scratch0, expected, fail); andr(scratch0, result, scratch1); // scratch1 is ~mask orr(scratch0, scratch0, new_val); - sc_w(scratch0, scratch0, aligned_addr, release); + store_conditional(scratch0, scratch0, aligned_addr, operand_size::int32, release); bnez(scratch0, fail); } @@ -3873,10 +3999,10 @@ void MacroAssembler::weak_cmpxchg_narrow_value(Register addr, Register expected, void MacroAssembler::cmpxchg(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result, bool result_as_bool) { - assert(size != int8 && size != int16, "unsupported operand size"); + assert((UseZacas && UseZabha) || (size != int8 && size != int16), "unsupported operand size"); assert_different_registers(addr, t0); assert_different_registers(expected, t0); assert_different_registers(new_val, t0); @@ -3934,10 +4060,10 @@ void MacroAssembler::cmpxchg(Register addr, Register expected, void MacroAssembler::weak_cmpxchg(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result) { - + assert((UseZacas && UseZabha) || (size != int8 && size != int16), "unsupported operand size"); assert_different_registers(addr, t0); assert_different_registers(expected, t0); assert_different_registers(new_val, t0); @@ -4010,7 +4136,7 @@ ATOMIC_XCHGU(xchgalwu, xchgalw) #undef ATOMIC_XCHGU void MacroAssembler::atomic_cas(Register prev, Register newv, Register addr, - enum operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release) { + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release) { switch (size) { case int64: amocas_d(prev, addr, newv, (Assembler::Aqrl)(acquire | release)); @@ -4022,6 +4148,12 @@ void MacroAssembler::atomic_cas(Register prev, Register newv, Register addr, amocas_w(prev, addr, newv, (Assembler::Aqrl)(acquire | release)); zext(prev, prev, 32); break; + case int16: + amocas_h(prev, addr, newv, (Assembler::Aqrl)(acquire | release)); + break; + case int8: + amocas_b(prev, addr, newv, (Assembler::Aqrl)(acquire | release)); + break; default: ShouldNotReachHere(); } @@ -4853,7 +4985,7 @@ address MacroAssembler::reloc_call(Address entry, Register tmp) { address MacroAssembler::ic_call(address entry, jint method_index) { RelocationHolder rh = virtual_call_Relocation::spec(pc(), method_index); - IncompressibleRegion ir(this); // relocations + IncompressibleScope scope(this); // relocations movptr(t0, (address)Universe::non_oop_word(), t1); assert_cond(entry != nullptr); return reloc_call(Address(entry, rh)); @@ -4866,7 +4998,7 @@ int MacroAssembler::ic_check_size() { } int MacroAssembler::ic_check(int end_alignment) { - IncompressibleRegion ir(this); + IncompressibleScope scope(this); Register receiver = j_rarg0; Register data = t0; @@ -5779,17 +5911,19 @@ void MacroAssembler::fill_words(Register base, Register cnt, Register value) { andi(t0, cnt, unroll - 1); sub(cnt, cnt, t0); - // align 8, so first sd n % 8 = mod, next loop sd 8 * n. shadd(base, t0, base, t1, 3); la(t1, entry); - slli(t0, t0, 2); // sd_inst_nums * 4; t0 is cnt % 8, so t1 = t1 - sd_inst_nums * 4, 4 is sizeof(inst) + slli(t0, t0, 2); sub(t1, t1, t0); jr(t1); bind(loop); addi(base, base, unroll * wordSize); - for (int i = -unroll; i < 0; i++) { - sd(value, Address(base, i * 8)); + { + IncompressibleScope scope(this); // Fixed length + for (int i = -unroll; i < 0; i++) { + sd(value, Address(base, i * 8)); + } } bind(entry); subi(cnt, cnt, unroll); @@ -5996,10 +6130,14 @@ void MacroAssembler::zero_memory(Register addr, Register len, Register tmp) { slli(t0, t0, 2); sub(t1, t1, t0); jr(t1); + bind(loop); sub(len, len, unroll); - for (int i = -unroll; i < 0; i++) { - sd(zr, Address(tmp, i * wordSize)); + { + IncompressibleScope scope(this); // Fixed length + for (int i = -unroll; i < 0; i++) { + sd(zr, Address(tmp, i * wordSize)); + } } bind(entry); add(tmp, tmp, unroll * wordSize); @@ -6362,10 +6500,17 @@ void MacroAssembler::lightweight_lock(Register basic_lock, Register obj, Registe ld(mark, Address(obj, oopDesc::mark_offset_in_bytes())); if (UseObjectMonitorTable) { - // Clear cache in case fast locking succeeds. + // Clear cache in case fast locking succeeds or we need to take the slow-path. sd(zr, Address(basic_lock, BasicObjectLock::lock_offset() + in_ByteSize((BasicLock::object_monitor_cache_offset_in_bytes())))); } + if (DiagnoseSyncOnValueBasedClasses != 0) { + load_klass(tmp1, obj); + lbu(tmp1, Address(tmp1, Klass::misc_flags_offset())); + test_bit(tmp1, tmp1, exact_log2(KlassFlags::_misc_is_value_based_class)); + bnez(tmp1, slow, /* is_far */ true); + } + // Check if the lock-stack is full. lwu(top, Address(xthread, JavaThread::lock_stack_top_offset())); mv(t, (unsigned)LockStack::end_offset()); diff --git a/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp b/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp index b390fb236c2..7fa7f931044 100644 --- a/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp +++ b/src/hotspot/cpu/riscv/macroAssembler_riscv.hpp @@ -44,7 +44,7 @@ class MacroAssembler: public Assembler { MacroAssembler(CodeBuffer* code) : Assembler(code) {} - void safepoint_poll(Label& slow_path, bool at_return, bool acquire, bool in_nmethod); + void safepoint_poll(Label& slow_path, bool at_return, bool acquire, bool in_nmethod, Register tmp_reg = t0); // Alignment int align(int modulus, int extra_offset = 0); @@ -657,11 +657,16 @@ class MacroAssembler: public Assembler { void cmov_gt(Register cmp1, Register cmp2, Register dst, Register src); void cmov_gtu(Register cmp1, Register cmp2, Register dst, Register src); + void cmov_cmp_fp_eq(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single); + void cmov_cmp_fp_ne(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single); + void cmov_cmp_fp_le(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single); + void cmov_cmp_fp_lt(FloatRegister cmp1, FloatRegister cmp2, Register dst, Register src, bool is_single); + public: // We try to follow risc-v asm menomics. // But as we don't layout a reachable GOT, // we often need to resort to movptr, li <48imm>. - // https://github.com/riscv-non-isa/riscv-asm-manual/blob/master/riscv-asm.md + // https://github.com/riscv-non-isa/riscv-asm-manual/blob/main/src/asm-manual.adoc // Hotspot only use the standard calling convention using x1/ra. // The alternative calling convection using x5/t0 is not used. @@ -744,7 +749,7 @@ class MacroAssembler: public Assembler { guarantee(rtype == relocInfo::internal_word_type, \ "only internal_word_type relocs make sense here"); \ relocate(InternalAddress(dest).rspec()); \ - IncompressibleRegion ir(this); /* relocations */ + IncompressibleScope scope(this); /* relocations */ #define INSN(NAME) \ void NAME(Register Rs1, Register Rs2, const address dest) { \ @@ -965,7 +970,7 @@ class MacroAssembler: public Assembler { guarantee(rtype == relocInfo::internal_word_type, \ "only internal_word_type relocs make sense here"); \ relocate(InternalAddress(dest).rspec()); \ - IncompressibleRegion ir(this); /* relocations */ + IncompressibleScope scope(this); /* relocations */ #define INSN(NAME) \ void NAME(Register Rd, address dest) { \ @@ -1182,26 +1187,26 @@ class MacroAssembler: public Assembler { void cmpxchgptr(Register oldv, Register newv, Register addr, Register tmp, Label &succeed, Label *fail); void cmpxchg(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result, bool result_as_bool = false); void weak_cmpxchg(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result); void cmpxchg_narrow_value_helper(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Register shift, Register mask, Register aligned_addr); void cmpxchg_narrow_value(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result, bool result_as_bool, Register tmp1, Register tmp2, Register tmp3); void weak_cmpxchg_narrow_value(Register addr, Register expected, Register new_val, - enum operand_size size, + Assembler::operand_size size, Assembler::Aqrl acquire, Assembler::Aqrl release, Register result, Register tmp1, Register tmp2, Register tmp3); @@ -1218,7 +1223,7 @@ class MacroAssembler: public Assembler { void atomic_xchgwu(Register prev, Register newv, Register addr); void atomic_xchgalwu(Register prev, Register newv, Register addr); - void atomic_cas(Register prev, Register newv, Register addr, enum operand_size size, + void atomic_cas(Register prev, Register newv, Register addr, Assembler::operand_size size, Assembler::Aqrl acquire = Assembler::relaxed, Assembler::Aqrl release = Assembler::relaxed); // Emit a far call/jump. Only invalidates the tmp register which @@ -1631,8 +1636,8 @@ class MacroAssembler: public Assembler { int bitset_to_regs(unsigned int bitset, unsigned char* regs); Address add_memory_helper(const Address dst, Register tmp); - void load_reserved(Register dst, Register addr, enum operand_size size, Assembler::Aqrl acquire); - void store_conditional(Register dst, Register new_val, Register addr, enum operand_size size, Assembler::Aqrl release); + void load_reserved(Register dst, Register addr, Assembler::operand_size size, Assembler::Aqrl acquire); + void store_conditional(Register dst, Register new_val, Register addr, Assembler::operand_size size, Assembler::Aqrl release); public: void lightweight_lock(Register basic_lock, Register obj, Register tmp1, Register tmp2, Register tmp3, Label& slow); diff --git a/src/hotspot/cpu/riscv/nativeInst_riscv.cpp b/src/hotspot/cpu/riscv/nativeInst_riscv.cpp index 6f20d54b222..31947b520d0 100644 --- a/src/hotspot/cpu/riscv/nativeInst_riscv.cpp +++ b/src/hotspot/cpu/riscv/nativeInst_riscv.cpp @@ -479,7 +479,7 @@ void NativeJump::patch_verified_entry(address entry, address verified_entry, add void NativeGeneralJump::insert_unconditional(address code_pos, address entry) { CodeBuffer cb(code_pos, instruction_size); MacroAssembler a(&cb); - Assembler::IncompressibleRegion ir(&a); // Fixed length: see NativeGeneralJump::get_instruction_size() + Assembler::IncompressibleScope scope(&a); // Fixed length: see NativeGeneralJump::get_instruction_size() int32_t offset = 0; a.movptr(t1, entry, offset, t0); // lui, lui, slli, add diff --git a/src/hotspot/cpu/riscv/nativeInst_riscv.hpp b/src/hotspot/cpu/riscv/nativeInst_riscv.hpp index 295e92bbc1b..d8f5fa57816 100644 --- a/src/hotspot/cpu/riscv/nativeInst_riscv.hpp +++ b/src/hotspot/cpu/riscv/nativeInst_riscv.hpp @@ -300,7 +300,7 @@ class NativeGeneralJump: public NativeJump { inline NativeGeneralJump* nativeGeneralJump_at(address addr) { assert_cond(addr != nullptr); NativeGeneralJump* jump = (NativeGeneralJump*)(addr); - debug_only(jump->verify();) + DEBUG_ONLY(jump->verify();) return jump; } diff --git a/src/hotspot/cpu/riscv/riscv.ad b/src/hotspot/cpu/riscv/riscv.ad index aca2f4dd488..e838ee184fb 100644 --- a/src/hotspot/cpu/riscv/riscv.ad +++ b/src/hotspot/cpu/riscv/riscv.ad @@ -1295,7 +1295,7 @@ uint MachBreakpointNode::size(PhaseRegAlloc *ra_) const { #endif void MachNopNode::emit(C2_MacroAssembler *masm, PhaseRegAlloc*) const { - Assembler::CompressibleRegion cr(masm); // nops shall be 2-byte under RVC for alignment purposes. + Assembler::CompressibleScope scope(masm); // nops shall be 2-byte under RVC for alignment purposes. for (int i = 0; i < _count; i++) { __ nop(); } @@ -1371,7 +1371,7 @@ void MachPrologNode::emit(C2_MacroAssembler *masm, PhaseRegAlloc *ra_) const { // insert a nop at the start of the prolog so we can patch in a // branch if we need to invalidate the method later { - Assembler::IncompressibleRegion ir(masm); // keep the nop as 4 bytes for patching. + Assembler::IncompressibleScope scope(masm); // keep the nop as 4 bytes for patching. MacroAssembler::assert_alignment(__ pc()); __ nop(); // 4 bytes } @@ -1596,7 +1596,8 @@ uint MachSpillCopyNode::implementation(C2_MacroAssembler *masm, PhaseRegAlloc *r __ unspill(as_VectorRegister(Matcher::_regEncode[dst_lo]), ra_->reg2offset(src_lo)); } else if (src_lo_rc == rc_vector && dst_lo_rc == rc_vector) { // vpr to vpr - __ vmv1r_v(as_VectorRegister(Matcher::_regEncode[dst_lo]), as_VectorRegister(Matcher::_regEncode[src_lo])); + __ vsetvli_helper(T_BYTE, MaxVectorSize); + __ vmv_v_v(as_VectorRegister(Matcher::_regEncode[dst_lo]), as_VectorRegister(Matcher::_regEncode[src_lo])); } else { ShouldNotReachHere(); } @@ -1614,7 +1615,8 @@ uint MachSpillCopyNode::implementation(C2_MacroAssembler *masm, PhaseRegAlloc *r __ unspill_vmask(as_VectorRegister(Matcher::_regEncode[dst_lo]), ra_->reg2offset(src_lo)); } else if (src_lo_rc == rc_vector && dst_lo_rc == rc_vector) { // vmask to vmask - __ vmv1r_v(as_VectorRegister(Matcher::_regEncode[dst_lo]), as_VectorRegister(Matcher::_regEncode[src_lo])); + __ vsetvli_helper(T_BYTE, MaxVectorSize >> 3); + __ vmv_v_v(as_VectorRegister(Matcher::_regEncode[dst_lo]), as_VectorRegister(Matcher::_regEncode[src_lo])); } else { ShouldNotReachHere(); } @@ -1752,7 +1754,7 @@ void BoxLockNode::format(PhaseRegAlloc *ra_, outputStream *st) const { #endif void BoxLockNode::emit(C2_MacroAssembler *masm, PhaseRegAlloc *ra_) const { - Assembler::IncompressibleRegion ir(masm); // Fixed length: see BoxLockNode::size() + Assembler::IncompressibleScope scope(masm); // Fixed length: see BoxLockNode::size() assert_cond(ra_ != nullptr); int offset = ra_->reg2offset(in_RegMask(0).find_first_elem()); @@ -1880,18 +1882,6 @@ bool Matcher::match_rule_supported(int opcode) { case Op_EncodeISOArray: return UseRVV; - // Current test shows that, it brings performance gain when MaxVectorSize >= 32, but brings - // regression when MaxVectorSize == 16. So only enable the intrinsic when MaxVectorSize >= 32. - case Op_RoundVF: - return UseRVV && MaxVectorSize >= 32; - - // For double, current test shows that even with MaxVectorSize == 32, there is still some regression. - // Although there is no hardware to verify it for now, from the trend of performance data on hardwares - // (with vlenb == 16 and 32 respectively), it's promising to bring better performance rather than - // regression for double when MaxVectorSize == 64+. So only enable the intrinsic when MaxVectorSize >= 64. - case Op_RoundVD: - return UseRVV && MaxVectorSize >= 64; - case Op_PopCountI: case Op_PopCountL: return UsePopCountInstruction; @@ -1914,8 +1904,6 @@ bool Matcher::match_rule_supported(int opcode) { case Op_FmaF: case Op_FmaD: - case Op_FmaVF: - case Op_FmaVD: return UseFMA; case Op_ConvHF2F: @@ -1930,9 +1918,15 @@ bool Matcher::match_rule_supported(int opcode) { case Op_MaxHF: case Op_MinHF: case Op_MulHF: - case Op_SubHF: case Op_SqrtHF: + case Op_SubHF: return UseZfh; + + case Op_CMoveF: + case Op_CMoveD: + case Op_CMoveP: + case Op_CMoveN: + return false; } return true; // Per default match rules are supported. @@ -1944,11 +1938,11 @@ const RegMask* Matcher::predicate_reg_mask(void) { // Vector calling convention not yet implemented. bool Matcher::supports_vector_calling_convention(void) { - return EnableVectorSupport && UseVectorStubs; + return EnableVectorSupport; } OptoRegPair Matcher::vector_return_value(uint ideal_reg) { - assert(EnableVectorSupport && UseVectorStubs, "sanity"); + assert(EnableVectorSupport, "sanity"); assert(ideal_reg == Op_VecA, "sanity"); // check more info at https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-cc.adoc int lo = V8_num; @@ -2310,42 +2304,6 @@ encode %{ } %} - enc_class riscv_enc_cmpxchgw(iRegINoSp res, memory mem, iRegI oldval, iRegI newval) %{ - __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int32, - /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, - /*result as bool*/ true); - %} - - enc_class riscv_enc_cmpxchgn(iRegINoSp res, memory mem, iRegI oldval, iRegI newval) %{ - __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::uint32, - /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, - /*result as bool*/ true); - %} - - enc_class riscv_enc_cmpxchg(iRegINoSp res, memory mem, iRegL oldval, iRegL newval) %{ - __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int64, - /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, - /*result as bool*/ true); - %} - - enc_class riscv_enc_cmpxchgw_acq(iRegINoSp res, memory mem, iRegI oldval, iRegI newval) %{ - __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int32, - /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, - /*result as bool*/ true); - %} - - enc_class riscv_enc_cmpxchgn_acq(iRegINoSp res, memory mem, iRegI oldval, iRegI newval) %{ - __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::uint32, - /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, - /*result as bool*/ true); - %} - - enc_class riscv_enc_cmpxchg_acq(iRegINoSp res, memory mem, iRegL oldval, iRegL newval) %{ - __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int64, - /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, - /*result as bool*/ true); - %} - // compare and branch instruction encodings enc_class riscv_enc_j(label lbl) %{ @@ -2395,7 +2353,7 @@ encode %{ %} enc_class riscv_enc_java_static_call(method meth) %{ - Assembler::IncompressibleRegion ir(masm); // Fixed length: see ret_addr_offset + Assembler::IncompressibleScope scope(masm); // Fixed length: see ret_addr_offset address addr = (address)$meth$$method; address call = nullptr; @@ -2442,7 +2400,7 @@ encode %{ %} enc_class riscv_enc_java_dynamic_call(method meth) %{ - Assembler::IncompressibleRegion ir(masm); // Fixed length: see ret_addr_offset + Assembler::IncompressibleScope scope(masm); // Fixed length: see ret_addr_offset int method_index = resolved_method_index(masm); address call = __ ic_call((address)$meth$$method, method_index); if (call == nullptr) { @@ -2461,7 +2419,7 @@ encode %{ %} enc_class riscv_enc_java_to_runtime(method meth) %{ - Assembler::IncompressibleRegion ir(masm); // Fixed length: see ret_addr_offset + Assembler::IncompressibleScope scope(masm); // Fixed length: see ret_addr_offset // Some calls to generated routines (arraycopy code) are scheduled by C2 // as runtime calls. if so we can call them using a far call (they will be @@ -5256,18 +5214,20 @@ instruct prefetchalloc( memory mem ) %{ // standard CompareAndSwapX when we are using barriers // these have higher priority than the rules selected by a predicate -instruct compareAndSwapB(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndSwapB_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) %{ + predicate(!UseZabha || !UseZacas); + match(Set res (CompareAndSwapB mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 10 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3, KILL cr); format %{ "cmpxchg $mem, $oldval, $newval\t# (byte) if $mem == $oldval then $mem <-- $newval\n\t" - "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapB" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapB_narrow" %} ins_encode %{ @@ -5279,18 +5239,42 @@ instruct compareAndSwapB(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R1 ins_pipe(pipe_slow); %} -instruct compareAndSwapS(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndSwapB(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ + predicate(UseZabha && UseZacas); + + match(Set res (CompareAndSwapB mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg $mem, $oldval, $newval\t# (byte) if $mem == $oldval then $mem <-- $newval\n\t" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapB" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int8, + Assembler::relaxed /* acquire */, Assembler::rl /* release */, $res$$Register, + true /* result as bool */); + %} + + ins_pipe(pipe_slow); +%} + +instruct compareAndSwapS_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +%{ + predicate(!UseZabha || !UseZacas); + match(Set res (CompareAndSwapS mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 11 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3, KILL cr); format %{ "cmpxchg $mem, $oldval, $newval\t# (short) if $mem == $oldval then $mem <-- $newval\n\t" - "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapS" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapS_narrow" %} ins_encode %{ @@ -5302,18 +5286,44 @@ instruct compareAndSwapS(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R1 ins_pipe(pipe_slow); %} +instruct compareAndSwapS(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) +%{ + predicate(UseZabha && UseZacas); + + match(Set res (CompareAndSwapS mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg $mem, $oldval, $newval\t# (short) if $mem == $oldval then $mem <-- $newval\n\t" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapS" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int16, + Assembler::relaxed /* acquire */, Assembler::rl /* release */, $res$$Register, + true /* result as bool */); + %} + + ins_pipe(pipe_slow); +%} + instruct compareAndSwapI(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ match(Set res (CompareAndSwapI mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 6 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $mem, $oldval, $newval\t# (int) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapI" %} - ins_encode(riscv_enc_cmpxchgw(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int32, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5322,14 +5332,18 @@ instruct compareAndSwapL(iRegINoSp res, indirect mem, iRegL oldval, iRegL newval %{ match(Set res (CompareAndSwapL mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 6 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $mem, $oldval, $newval\t# (long) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapL" %} - ins_encode(riscv_enc_cmpxchg(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int64, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5340,14 +5354,18 @@ instruct compareAndSwapP(iRegINoSp res, indirect mem, iRegP oldval, iRegP newval match(Set res (CompareAndSwapP mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 6 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $mem, $oldval, $newval\t# (ptr) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapP" %} - ins_encode(riscv_enc_cmpxchg(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int64, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5355,35 +5373,40 @@ instruct compareAndSwapP(iRegINoSp res, indirect mem, iRegP oldval, iRegP newval instruct compareAndSwapN(iRegINoSp res, indirect mem, iRegN oldval, iRegN newval) %{ predicate(n->as_LoadStore()->barrier_data() == 0); + match(Set res (CompareAndSwapN mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 8 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $mem, $oldval, $newval\t# (narrow oop) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapN" %} - ins_encode(riscv_enc_cmpxchgn(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::uint32, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} // alternative CompareAndSwapX when we are eliding barriers -instruct compareAndSwapBAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndSwapBAcq_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) %{ - predicate(needs_acquiring_load_reserved(n)); + predicate((!UseZabha || !UseZacas) && needs_acquiring_load_reserved(n)); match(Set res (CompareAndSwapB mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 10 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ "cmpxchg_acq $mem, $oldval, $newval\t# (byte) if $mem == $oldval then $mem <-- $newval\n\t" - "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapBAcq" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapBAcq_narrow" %} ins_encode %{ @@ -5395,20 +5418,42 @@ instruct compareAndSwapBAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI ins_pipe(pipe_slow); %} -instruct compareAndSwapSAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndSwapBAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ - predicate(needs_acquiring_load_reserved(n)); + predicate((UseZabha && UseZacas) && needs_acquiring_load_reserved(n)); + + match(Set res (CompareAndSwapB mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg $mem, $oldval, $newval\t# (byte) if $mem == $oldval then $mem <-- $newval\n\t" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapBAcq" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int8, + Assembler::aq /* acquire */, Assembler::rl /* release */, $res$$Register, + true /* result as bool */); + %} + + ins_pipe(pipe_slow); +%} + +instruct compareAndSwapSAcq_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +%{ + predicate((!UseZabha || !UseZacas) && needs_acquiring_load_reserved(n)); match(Set res (CompareAndSwapS mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 11 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ "cmpxchg_acq $mem, $oldval, $newval\t# (short) if $mem == $oldval then $mem <-- $newval\n\t" - "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapSAcq" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapSAcq_narrow" %} ins_encode %{ @@ -5420,20 +5465,46 @@ instruct compareAndSwapSAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI ins_pipe(pipe_slow); %} +instruct compareAndSwapSAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) +%{ + predicate((UseZabha && UseZacas) && needs_acquiring_load_reserved(n)); + + match(Set res (CompareAndSwapS mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg $mem, $oldval, $newval\t# (short) if $mem == $oldval then $mem <-- $newval\n\t" + "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapSAcq" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int16, + Assembler::aq /* acquire */, Assembler::rl /* release */, $res$$Register, + true /* result as bool */); + %} + + ins_pipe(pipe_slow); +%} + instruct compareAndSwapIAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ predicate(needs_acquiring_load_reserved(n)); match(Set res (CompareAndSwapI mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 6 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $mem, $oldval, $newval\t# (int) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapIAcq" %} - ins_encode(riscv_enc_cmpxchgw_acq(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int32, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5444,14 +5515,18 @@ instruct compareAndSwapLAcq(iRegINoSp res, indirect mem, iRegL oldval, iRegL new match(Set res (CompareAndSwapL mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 6 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $mem, $oldval, $newval\t# (long) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapLAcq" %} - ins_encode(riscv_enc_cmpxchg_acq(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int64, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5462,14 +5537,18 @@ instruct compareAndSwapPAcq(iRegINoSp res, indirect mem, iRegP oldval, iRegP new match(Set res (CompareAndSwapP mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 6 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $mem, $oldval, $newval\t# (ptr) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapPAcq" %} - ins_encode(riscv_enc_cmpxchg_acq(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int64, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5480,14 +5559,18 @@ instruct compareAndSwapNAcq(iRegINoSp res, indirect mem, iRegN oldval, iRegN new match(Set res (CompareAndSwapN mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + ALU_COST * 8 + BRANCH_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $mem, $oldval, $newval\t# (narrow oop) if $mem == $oldval then $mem <-- $newval\n\t" "mv $res, $res == $oldval\t# $res <-- ($res == $oldval ? 1 : 0), #@compareAndSwapNAcq" %} - ins_encode(riscv_enc_cmpxchgn_acq(res, mem, oldval, newval)); + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::uint32, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register, + /*result as bool*/ true); + %} ins_pipe(pipe_slow); %} @@ -5498,17 +5581,19 @@ instruct compareAndSwapNAcq(iRegINoSp res, indirect mem, iRegN oldval, iRegN new // no trailing StoreLoad barrier emitted by C2. Unfortunately we // can't check the type of memory ordering here, so we always emit a // sc_d(w) with rl bit set. -instruct compareAndExchangeB(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndExchangeB_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) %{ + predicate(!UseZabha || !UseZacas); + match(Set res (CompareAndExchangeB mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST * 5); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ - "cmpxchg $res = $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeB" + "cmpxchg $res = $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeB_narrow" %} ins_encode %{ @@ -5520,17 +5605,39 @@ instruct compareAndExchangeB(iRegINoSp res, indirect mem, iRegI_R12 oldval, iReg ins_pipe(pipe_slow); %} -instruct compareAndExchangeS(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndExchangeB(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ + predicate(UseZabha && UseZacas); + + match(Set res (CompareAndExchangeB mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg $res = $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeB" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int8, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + +instruct compareAndExchangeS_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +%{ + predicate(!UseZabha || !UseZacas); + match(Set res (CompareAndExchangeS mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST * 6); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ - "cmpxchg $res = $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeS" + "cmpxchg $res = $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeS_narrow" %} ins_encode %{ @@ -5542,13 +5649,31 @@ instruct compareAndExchangeS(iRegINoSp res, indirect mem, iRegI_R12 oldval, iReg ins_pipe(pipe_slow); %} +instruct compareAndExchangeS(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) +%{ + predicate(UseZabha && UseZacas); + + match(Set res (CompareAndExchangeS mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg $res = $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeS" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int16, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + instruct compareAndExchangeI(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ match(Set res (CompareAndExchangeI mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); - - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $res = $mem, $oldval, $newval\t# (int, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeI" @@ -5566,9 +5691,7 @@ instruct compareAndExchangeL(iRegLNoSp res, indirect mem, iRegL oldval, iRegL ne %{ match(Set res (CompareAndExchangeL mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); - - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $res = $mem, $oldval, $newval\t# (long, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeL" @@ -5585,11 +5708,10 @@ instruct compareAndExchangeL(iRegLNoSp res, indirect mem, iRegL oldval, iRegL ne instruct compareAndExchangeN(iRegNNoSp res, indirect mem, iRegN oldval, iRegN newval) %{ predicate(n->as_LoadStore()->barrier_data() == 0); - match(Set res (CompareAndExchangeN mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST * 3); + match(Set res (CompareAndExchangeN mem (Binary oldval newval))); - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $res = $mem, $oldval, $newval\t# (narrow oop, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeN" @@ -5606,11 +5728,10 @@ instruct compareAndExchangeN(iRegNNoSp res, indirect mem, iRegN oldval, iRegN ne instruct compareAndExchangeP(iRegPNoSp res, indirect mem, iRegP oldval, iRegP newval) %{ predicate(n->as_LoadStore()->barrier_data() == 0); - match(Set res (CompareAndExchangeP mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); + match(Set res (CompareAndExchangeP mem (Binary oldval newval))); - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg $res = $mem, $oldval, $newval\t# (ptr, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeP" @@ -5624,19 +5745,19 @@ instruct compareAndExchangeP(iRegPNoSp res, indirect mem, iRegP oldval, iRegP ne ins_pipe(pipe_slow); %} -instruct compareAndExchangeBAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndExchangeBAcq_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) %{ - predicate(needs_acquiring_load_reserved(n)); + predicate((!UseZabha || !UseZacas) && needs_acquiring_load_reserved(n)); match(Set res (CompareAndExchangeB mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST * 5); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ - "cmpxchg_acq $res = $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeBAcq" + "cmpxchg_acq $res = $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeBAcq_narrow" %} ins_encode %{ @@ -5648,19 +5769,39 @@ instruct compareAndExchangeBAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, i ins_pipe(pipe_slow); %} -instruct compareAndExchangeSAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct compareAndExchangeBAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ - predicate(needs_acquiring_load_reserved(n)); + predicate((UseZabha && UseZacas) && needs_acquiring_load_reserved(n)); + + match(Set res (CompareAndExchangeB mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg_acq $res = $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeBAcq" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int8, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + +instruct compareAndExchangeSAcq_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +%{ + predicate((!UseZabha || !UseZacas) && needs_acquiring_load_reserved(n)); match(Set res (CompareAndExchangeS mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST * 6); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ - "cmpxchg_acq $res = $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeSAcq" + "cmpxchg_acq $res = $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeSAcq_narrow" %} ins_encode %{ @@ -5672,15 +5813,33 @@ instruct compareAndExchangeSAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, i ins_pipe(pipe_slow); %} +instruct compareAndExchangeSAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) +%{ + predicate((UseZabha && UseZacas) && needs_acquiring_load_reserved(n)); + + match(Set res (CompareAndExchangeS mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "cmpxchg_acq $res = $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeSAcq" + %} + + ins_encode %{ + __ cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int16, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + instruct compareAndExchangeIAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ predicate(needs_acquiring_load_reserved(n)); match(Set res (CompareAndExchangeI mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); - - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $res = $mem, $oldval, $newval\t# (int, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeIAcq" @@ -5700,9 +5859,7 @@ instruct compareAndExchangeLAcq(iRegLNoSp res, indirect mem, iRegL oldval, iRegL match(Set res (CompareAndExchangeL mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); - - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $res = $mem, $oldval, $newval\t# (long, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeLAcq" @@ -5722,9 +5879,7 @@ instruct compareAndExchangeNAcq(iRegNNoSp res, indirect mem, iRegN oldval, iRegN match(Set res (CompareAndExchangeN mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); - - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $res = $mem, $oldval, $newval\t# (narrow oop, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangeNAcq" @@ -5744,9 +5899,7 @@ instruct compareAndExchangePAcq(iRegPNoSp res, indirect mem, iRegP oldval, iRegP match(Set res (CompareAndExchangeP mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 3 + ALU_COST); - - effect(TEMP_DEF res); + ins_cost(2 * VOLATILE_REF_COST); format %{ "cmpxchg_acq $res = $mem, $oldval, $newval\t# (ptr, weak) if $mem == $oldval then $mem <-- $newval, #@compareAndExchangePAcq" @@ -5760,18 +5913,20 @@ instruct compareAndExchangePAcq(iRegPNoSp res, indirect mem, iRegP oldval, iRegP ins_pipe(pipe_slow); %} -instruct weakCompareAndSwapB(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct weakCompareAndSwapB_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) %{ + predicate(!UseZabha || !UseZacas); + match(Set res (WeakCompareAndSwapB mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 6); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ "weak_cmpxchg $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval\n\t" - "# $res == 1 when success, #@weakCompareAndSwapB" + "# $res == 1 when success, #@weakCompareAndSwapB_narrow" %} ins_encode %{ @@ -5783,18 +5938,41 @@ instruct weakCompareAndSwapB(iRegINoSp res, indirect mem, iRegI_R12 oldval, iReg ins_pipe(pipe_slow); %} -instruct weakCompareAndSwapS(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct weakCompareAndSwapB(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ + predicate(UseZabha && UseZacas); + + match(Set res (WeakCompareAndSwapB mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "weak_cmpxchg $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval\n\t" + "# $res == 1 when success, #@weakCompareAndSwapB" + %} + + ins_encode %{ + __ weak_cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int8, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + +instruct weakCompareAndSwapS_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +%{ + predicate(!UseZabha || !UseZacas); + match(Set res (WeakCompareAndSwapS mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 7); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ "weak_cmpxchg $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval\n\t" - "# $res == 1 when success, #@weakCompareAndSwapS" + "# $res == 1 when success, #@weakCompareAndSwapS_narrow" %} ins_encode %{ @@ -5806,11 +5984,32 @@ instruct weakCompareAndSwapS(iRegINoSp res, indirect mem, iRegI_R12 oldval, iReg ins_pipe(pipe_slow); %} +instruct weakCompareAndSwapS(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) +%{ + predicate(UseZabha && UseZacas); + + match(Set res (WeakCompareAndSwapS mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "weak_cmpxchg $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval\n\t" + "# $res == 1 when success, #@weakCompareAndSwapS" + %} + + ins_encode %{ + __ weak_cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int16, + /*acquire*/ Assembler::relaxed, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + instruct weakCompareAndSwapI(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ match(Set res (WeakCompareAndSwapI mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 2); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg $mem, $oldval, $newval\t# (int, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -5829,7 +6028,7 @@ instruct weakCompareAndSwapL(iRegINoSp res, indirect mem, iRegL oldval, iRegL ne %{ match(Set res (WeakCompareAndSwapL mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 2); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg $mem, $oldval, $newval\t# (long, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -5847,9 +6046,10 @@ instruct weakCompareAndSwapL(iRegINoSp res, indirect mem, iRegL oldval, iRegL ne instruct weakCompareAndSwapN(iRegINoSp res, indirect mem, iRegN oldval, iRegN newval) %{ predicate(n->as_LoadStore()->barrier_data() == 0); + match(Set res (WeakCompareAndSwapN mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg $mem, $oldval, $newval\t# (narrow oop, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -5867,9 +6067,10 @@ instruct weakCompareAndSwapN(iRegINoSp res, indirect mem, iRegN oldval, iRegN ne instruct weakCompareAndSwapP(iRegINoSp res, indirect mem, iRegP oldval, iRegP newval) %{ predicate(n->as_LoadStore()->barrier_data() == 0); + match(Set res (WeakCompareAndSwapP mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 2); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg $mem, $oldval, $newval\t# (ptr, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -5884,20 +6085,20 @@ instruct weakCompareAndSwapP(iRegINoSp res, indirect mem, iRegP oldval, iRegP ne ins_pipe(pipe_slow); %} -instruct weakCompareAndSwapBAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct weakCompareAndSwapBAcq_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) %{ - predicate(needs_acquiring_load_reserved(n)); + predicate((!UseZabha || !UseZacas) && needs_acquiring_load_reserved(n)); match(Set res (WeakCompareAndSwapB mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 6); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ "weak_cmpxchg_acq $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval\n\t" - "# $res == 1 when success, #@weakCompareAndSwapBAcq" + "# $res == 1 when success, #@weakCompareAndSwapBAcq_narrow" %} ins_encode %{ @@ -5909,20 +6110,41 @@ instruct weakCompareAndSwapBAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, i ins_pipe(pipe_slow); %} -instruct weakCompareAndSwapSAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, - iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +instruct weakCompareAndSwapBAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ - predicate(needs_acquiring_load_reserved(n)); + predicate((UseZabha && UseZacas) && needs_acquiring_load_reserved(n)); + + match(Set res (WeakCompareAndSwapB mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "weak_cmpxchg_acq $mem, $oldval, $newval\t# (byte, weak) if $mem == $oldval then $mem <-- $newval\n\t" + "# $res == 1 when success, #@weakCompareAndSwapBAcq" + %} + + ins_encode %{ + __ weak_cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int8, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + +instruct weakCompareAndSwapSAcq_narrow(iRegINoSp res, indirect mem, iRegI_R12 oldval, iRegI_R13 newval, + iRegINoSp tmp1, iRegINoSp tmp2, iRegINoSp tmp3, rFlagsReg cr) +%{ + predicate((!UseZabha || !UseZacas) && needs_acquiring_load_reserved(n)); match(Set res (WeakCompareAndSwapS mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 7); + ins_cost(2 * VOLATILE_REF_COST); effect(TEMP_DEF res, KILL cr, USE_KILL oldval, USE_KILL newval, TEMP tmp1, TEMP tmp2, TEMP tmp3); format %{ "weak_cmpxchg_acq $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval\n\t" - "# $res == 1 when success, #@weakCompareAndSwapSAcq" + "# $res == 1 when success, #@weakCompareAndSwapSAcq_narrow" %} ins_encode %{ @@ -5934,13 +6156,34 @@ instruct weakCompareAndSwapSAcq(iRegINoSp res, indirect mem, iRegI_R12 oldval, i ins_pipe(pipe_slow); %} +instruct weakCompareAndSwapSAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) +%{ + predicate((UseZabha && UseZacas) && needs_acquiring_load_reserved(n)); + + match(Set res (WeakCompareAndSwapS mem (Binary oldval newval))); + + ins_cost(2 * VOLATILE_REF_COST); + + format %{ + "weak_cmpxchg_acq $mem, $oldval, $newval\t# (short, weak) if $mem == $oldval then $mem <-- $newval\n\t" + "# $res == 1 when success, #@weakCompareAndSwapSAcq" + %} + + ins_encode %{ + __ weak_cmpxchg(as_Register($mem$$base), $oldval$$Register, $newval$$Register, Assembler::int16, + /*acquire*/ Assembler::aq, /*release*/ Assembler::rl, $res$$Register); + %} + + ins_pipe(pipe_slow); +%} + instruct weakCompareAndSwapIAcq(iRegINoSp res, indirect mem, iRegI oldval, iRegI newval) %{ predicate(needs_acquiring_load_reserved(n)); match(Set res (WeakCompareAndSwapI mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 2); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg_acq $mem, $oldval, $newval\t# (int, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -5961,7 +6204,7 @@ instruct weakCompareAndSwapLAcq(iRegINoSp res, indirect mem, iRegL oldval, iRegL match(Set res (WeakCompareAndSwapL mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 2); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg_acq $mem, $oldval, $newval\t# (long, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -5982,7 +6225,7 @@ instruct weakCompareAndSwapNAcq(iRegINoSp res, indirect mem, iRegN oldval, iRegN match(Set res (WeakCompareAndSwapN mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 4); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg_acq $mem, $oldval, $newval\t# (narrow oop, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -6003,7 +6246,7 @@ instruct weakCompareAndSwapPAcq(iRegINoSp res, indirect mem, iRegP oldval, iRegP match(Set res (WeakCompareAndSwapP mem (Binary oldval newval))); - ins_cost(LOAD_COST + STORE_COST + BRANCH_COST * 2 + ALU_COST * 2); + ins_cost(2 * VOLATILE_REF_COST); format %{ "weak_cmpxchg_acq $mem, $oldval, $newval\t# (ptr, weak) if $mem == $oldval then $mem <-- $newval\n\t" @@ -6445,7 +6688,6 @@ instruct addI_reg_imm(iRegINoSp dst, iRegIorL2I src1, immIAdd src2) %{ format %{ "addiw $dst, $src1, $src2\t#@addI_reg_imm" %} ins_encode %{ - int32_t con = (int32_t)$src2$$constant; __ addiw(as_Register($dst$$reg), as_Register($src1$$reg), $src2$$constant); @@ -6507,7 +6749,6 @@ instruct addP_reg_imm(iRegPNoSp dst, iRegP src1, immLAdd src2) %{ format %{ "addi $dst, $src1, $src2\t# ptr, #@addP_reg_imm" %} ins_encode %{ - // src2 is imm, so actually call the addi __ addi(as_Register($dst$$reg), as_Register($src1$$reg), $src2$$constant); @@ -6829,7 +7070,7 @@ instruct UmodL(iRegLNoSp dst, iRegL src1, iRegL src2) %{ // Integer Shifts // Shift Left Register -// In RV64I, only the low 5 bits of src2 are considered for the shift amount +// Only the low 5 bits of src2 are considered for the shift amount, all other bits are ignored. instruct lShiftI_reg_reg(iRegINoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{ match(Set dst (LShiftI src1 src2)); ins_cost(ALU_COST); @@ -6862,7 +7103,7 @@ instruct lShiftI_reg_imm(iRegINoSp dst, iRegIorL2I src1, immI src2) %{ %} // Shift Right Logical Register -// In RV64I, only the low 5 bits of src2 are considered for the shift amount +// Only the low 5 bits of src2 are considered for the shift amount, all other bits are ignored. instruct urShiftI_reg_reg(iRegINoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{ match(Set dst (URShiftI src1 src2)); ins_cost(ALU_COST); @@ -6895,7 +7136,7 @@ instruct urShiftI_reg_imm(iRegINoSp dst, iRegIorL2I src1, immI src2) %{ %} // Shift Right Arithmetic Register -// In RV64I, only the low 5 bits of src2 are considered for the shift amount +// Only the low 5 bits of src2 are considered for the shift amount, all other bits are ignored. instruct rShiftI_reg_reg(iRegINoSp dst, iRegIorL2I src1, iRegIorL2I src2) %{ match(Set dst (RShiftI src1 src2)); ins_cost(ALU_COST); @@ -6930,7 +7171,7 @@ instruct rShiftI_reg_imm(iRegINoSp dst, iRegIorL2I src1, immI src2) %{ // Long Shifts // Shift Left Register -// In RV64I, only the low 6 bits of src2 are considered for the shift amount +// Only the low 6 bits of src2 are considered for the shift amount, all other bits are ignored. instruct lShiftL_reg_reg(iRegLNoSp dst, iRegL src1, iRegIorL2I src2) %{ match(Set dst (LShiftL src1 src2)); @@ -6965,7 +7206,7 @@ instruct lShiftL_reg_imm(iRegLNoSp dst, iRegL src1, immI src2) %{ %} // Shift Right Logical Register -// In RV64I, only the low 6 bits of src2 are considered for the shift amount +// Only the low 6 bits of src2 are considered for the shift amount, all other bits are ignored. instruct urShiftL_reg_reg(iRegLNoSp dst, iRegL src1, iRegIorL2I src2) %{ match(Set dst (URShiftL src1 src2)); @@ -7018,7 +7259,7 @@ instruct urShiftP_reg_imm(iRegLNoSp dst, iRegP src1, immI src2) %{ %} // Shift Right Arithmetic Register -// In RV64I, only the low 6 bits of src2 are considered for the shift amount +// Only the low 6 bits of src2 are considered for the shift amount, all other bits are ignored. instruct rShiftL_reg_reg(iRegLNoSp dst, iRegL src1, iRegIorL2I src2) %{ match(Set dst (RShiftL src1 src2)); @@ -9940,12 +10181,15 @@ instruct far_cmpP_narrowOop_imm0_branch(cmpOpEqNe cmp, iRegN op1, immP0 zero, la // ============================================================================ // Conditional Move Instructions + +// --------- CMoveI --------- + instruct cmovI_cmpI(iRegINoSp dst, iRegI src, iRegI op1, iRegI op2, cmpOp cop) %{ match(Set dst (CMoveI (Binary cop (CmpI op1 op2)) (Binary dst src))); ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpI\n\t" + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpI\n\t" %} ins_encode %{ @@ -9962,7 +10206,7 @@ instruct cmovI_cmpU(iRegINoSp dst, iRegI src, iRegI op1, iRegI op2, cmpOpU cop) ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpU\n\t" + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpU\n\t" %} ins_encode %{ @@ -9979,7 +10223,7 @@ instruct cmovI_cmpL(iRegINoSp dst, iRegI src, iRegL op1, iRegL op2, cmpOp cop) % ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpL\n\t" + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpL\n\t" %} ins_encode %{ @@ -9996,7 +10240,7 @@ instruct cmovI_cmpUL(iRegINoSp dst, iRegI src, iRegL op1, iRegL op2, cmpOpU cop) ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpUL\n\t" + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpUL\n\t" %} ins_encode %{ @@ -10008,12 +10252,46 @@ instruct cmovI_cmpUL(iRegINoSp dst, iRegI src, iRegL op1, iRegL op2, cmpOpU cop) ins_pipe(pipe_class_compare); %} +instruct cmovI_cmpF(iRegINoSp dst, iRegI src, fRegF op1, fRegF op2, cmpOp cop) %{ + match(Set dst (CMoveI (Binary cop (CmpF op1 op2)) (Binary dst src))); + ins_cost(ALU_COST + BRANCH_COST); + + format %{ + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpF\n\t" + %} + + ins_encode %{ + __ enc_cmove_cmp_fp($cop$$cmpcode, + as_FloatRegister($op1$$reg), as_FloatRegister($op2$$reg), + as_Register($dst$$reg), as_Register($src$$reg), true /* is_single */); + %} + + ins_pipe(pipe_class_compare); +%} + +instruct cmovI_cmpD(iRegINoSp dst, iRegI src, fRegD op1, fRegD op2, cmpOp cop) %{ + match(Set dst (CMoveI (Binary cop (CmpD op1 op2)) (Binary dst src))); + ins_cost(ALU_COST + BRANCH_COST); + + format %{ + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpD\n\t" + %} + + ins_encode %{ + __ enc_cmove_cmp_fp($cop$$cmpcode | C2_MacroAssembler::double_branch_mask, + as_FloatRegister($op1$$reg), as_FloatRegister($op2$$reg), + as_Register($dst$$reg), as_Register($src$$reg), false /* is_single */); + %} + + ins_pipe(pipe_class_compare); +%} + instruct cmovI_cmpN(iRegINoSp dst, iRegI src, iRegN op1, iRegN op2, cmpOpU cop) %{ match(Set dst (CMoveI (Binary cop (CmpN op1 op2)) (Binary dst src))); ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpN\n\t" + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpN\n\t" %} ins_encode %{ @@ -10030,7 +10308,7 @@ instruct cmovI_cmpP(iRegINoSp dst, iRegI src, iRegP op1, iRegP op2, cmpOpU cop) ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpP\n\t" + "CMoveI $dst, ($op1 $cop $op2), $dst, $src\t#@cmovI_cmpP\n\t" %} ins_encode %{ @@ -10042,12 +10320,14 @@ instruct cmovI_cmpP(iRegINoSp dst, iRegI src, iRegP op1, iRegP op2, cmpOpU cop) ins_pipe(pipe_class_compare); %} +// --------- CMoveL --------- + instruct cmovL_cmpL(iRegLNoSp dst, iRegL src, iRegL op1, iRegL op2, cmpOp cop) %{ match(Set dst (CMoveL (Binary cop (CmpL op1 op2)) (Binary dst src))); ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpL\n\t" + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpL\n\t" %} ins_encode %{ @@ -10064,7 +10344,7 @@ instruct cmovL_cmpUL(iRegLNoSp dst, iRegL src, iRegL op1, iRegL op2, cmpOpU cop) ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpUL\n\t" + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpUL\n\t" %} ins_encode %{ @@ -10081,7 +10361,7 @@ instruct cmovL_cmpI(iRegLNoSp dst, iRegL src, iRegI op1, iRegI op2, cmpOp cop) % ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpI\n\t" + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpI\n\t" %} ins_encode %{ @@ -10098,7 +10378,7 @@ instruct cmovL_cmpU(iRegLNoSp dst, iRegL src, iRegI op1, iRegI op2, cmpOpU cop) ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpU\n\t" + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpU\n\t" %} ins_encode %{ @@ -10110,12 +10390,46 @@ instruct cmovL_cmpU(iRegLNoSp dst, iRegL src, iRegI op1, iRegI op2, cmpOpU cop) ins_pipe(pipe_class_compare); %} +instruct cmovL_cmpF(iRegLNoSp dst, iRegL src, fRegF op1, fRegF op2, cmpOp cop) %{ + match(Set dst (CMoveL (Binary cop (CmpF op1 op2)) (Binary dst src))); + ins_cost(ALU_COST + BRANCH_COST); + + format %{ + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpF\n\t" + %} + + ins_encode %{ + __ enc_cmove_cmp_fp($cop$$cmpcode, + as_FloatRegister($op1$$reg), as_FloatRegister($op2$$reg), + as_Register($dst$$reg), as_Register($src$$reg), true /* is_single */); + %} + + ins_pipe(pipe_class_compare); +%} + +instruct cmovL_cmpD(iRegLNoSp dst, iRegL src, fRegD op1, fRegD op2, cmpOp cop) %{ + match(Set dst (CMoveL (Binary cop (CmpD op1 op2)) (Binary dst src))); + ins_cost(ALU_COST + BRANCH_COST); + + format %{ + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpD\n\t" + %} + + ins_encode %{ + __ enc_cmove_cmp_fp($cop$$cmpcode | C2_MacroAssembler::double_branch_mask, + as_FloatRegister($op1$$reg), as_FloatRegister($op2$$reg), + as_Register($dst$$reg), as_Register($src$$reg), false /* is_single */); + %} + + ins_pipe(pipe_class_compare); +%} + instruct cmovL_cmpN(iRegLNoSp dst, iRegL src, iRegN op1, iRegN op2, cmpOpU cop) %{ match(Set dst (CMoveL (Binary cop (CmpN op1 op2)) (Binary dst src))); ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpN\n\t" + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpN\n\t" %} ins_encode %{ @@ -10132,7 +10446,7 @@ instruct cmovL_cmpP(iRegLNoSp dst, iRegL src, iRegP op1, iRegP op2, cmpOpU cop) ins_cost(ALU_COST + BRANCH_COST); format %{ - "CMove $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpP\n\t" + "CMoveL $dst, ($op1 $cop $op2), $dst, $src\t#@cmovL_cmpP\n\t" %} ins_encode %{ diff --git a/src/hotspot/cpu/riscv/riscv_b.ad b/src/hotspot/cpu/riscv/riscv_b.ad index ed9fca13a1b..beac10ec03d 100644 --- a/src/hotspot/cpu/riscv/riscv_b.ad +++ b/src/hotspot/cpu/riscv/riscv_b.ad @@ -25,7 +25,8 @@ // RISCV Bit-Manipulation Extension Architecture Description File -instruct rorI_imm_b(iRegINoSp dst, iRegI src, immI shift) %{ +// Rotate Right Word Immediate +instruct rorI_imm_b(iRegINoSp dst, iRegIorL2I src, immI shift) %{ predicate(UseZbb); match(Set dst (RotateRight src shift)); @@ -39,6 +40,7 @@ instruct rorI_imm_b(iRegINoSp dst, iRegI src, immI shift) %{ ins_pipe(ialu_reg_shift); %} +// Rotate Right Immediate instruct rorL_imm_b(iRegLNoSp dst, iRegL src, immI shift) %{ predicate(UseZbb); match(Set dst (RotateRight src shift)); @@ -53,7 +55,9 @@ instruct rorL_imm_b(iRegLNoSp dst, iRegL src, immI shift) %{ ins_pipe(ialu_reg_shift); %} -instruct rorI_reg_b(iRegINoSp dst, iRegI src, iRegI shift) %{ +// Rotate Right Word Register +// Only the low 5 bits of shift value are used, all other bits are ignored. +instruct rorI_reg_b(iRegINoSp dst, iRegIorL2I src, iRegIorL2I shift) %{ predicate(UseZbb); match(Set dst (RotateRight src shift)); @@ -65,7 +69,9 @@ instruct rorI_reg_b(iRegINoSp dst, iRegI src, iRegI shift) %{ ins_pipe(ialu_reg_reg); %} -instruct rorL_reg_b(iRegLNoSp dst, iRegL src, iRegI shift) %{ +// Rotate Right Register +// Only the low 6 bits of shift value are used, all other bits are ignored. +instruct rorL_reg_b(iRegLNoSp dst, iRegL src, iRegIorL2I shift) %{ predicate(UseZbb); match(Set dst (RotateRight src shift)); @@ -77,7 +83,9 @@ instruct rorL_reg_b(iRegLNoSp dst, iRegL src, iRegI shift) %{ ins_pipe(ialu_reg_reg); %} -instruct rolI_reg_b(iRegINoSp dst, iRegI src, iRegI shift) %{ +// Rotate Left Word Register +// Only the low 5 bits of shift value are used, all other bits are ignored. +instruct rolI_reg_b(iRegINoSp dst, iRegIorL2I src, iRegIorL2I shift) %{ predicate(UseZbb); match(Set dst (RotateLeft src shift)); @@ -89,7 +97,9 @@ instruct rolI_reg_b(iRegINoSp dst, iRegI src, iRegI shift) %{ ins_pipe(ialu_reg_reg); %} -instruct rolL_reg_b(iRegLNoSp dst, iRegL src, iRegI shift) %{ +// Rotate Left Register +// Only the low 6 bits of shift value are used, all other bits are ignored. +instruct rolL_reg_b(iRegLNoSp dst, iRegL src, iRegIorL2I shift) %{ predicate(UseZbb); match(Set dst (RotateLeft src shift)); diff --git a/src/hotspot/cpu/riscv/riscv_v.ad b/src/hotspot/cpu/riscv/riscv_v.ad index 9b135215b3d..8b5759ce11c 100644 --- a/src/hotspot/cpu/riscv/riscv_v.ad +++ b/src/hotspot/cpu/riscv/riscv_v.ad @@ -89,13 +89,12 @@ source %{ return UseZvbb; case Op_LoadVectorGather: case Op_LoadVectorGatherMasked: + case Op_StoreVectorScatter: + case Op_StoreVectorScatterMasked: if (is_subword_type(bt)) { return false; } break; - case Op_VectorCastHF2F: - case Op_VectorCastF2HF: - return UseZvfh; case Op_VectorLoadShuffle: case Op_VectorRearrange: // vlen >= 4 is required, because min vector size for byte is 4 on riscv, @@ -111,6 +110,32 @@ source %{ if (vlen < 4) { return false; } + case Op_VectorCastHF2F: + case Op_VectorCastF2HF: + case Op_AddVHF: + case Op_DivVHF: + case Op_MaxVHF: + case Op_MinVHF: + case Op_MulVHF: + case Op_SqrtVHF: + case Op_SubVHF: + return UseZvfh; + case Op_FmaVHF: + return UseZvfh && UseFMA; + case Op_FmaVF: + case Op_FmaVD: + return UseFMA; + + // For float, current test shows that, it brings performance gain when vlen >= 8, but brings + // regression when vlen == 4. So only enable this intrinsic when vlen >= 8. + // For double, current test shows that even with vlen == 4, there is still some regression. + // Although there is no hardware to verify it, from the trend of performance data on hardwares + // (with vlen == 2 and 4 respectively), it's promising to bring better performance rather than + // regression for double when vlen == 8. So only enable this intrinsic when vlen >= 8. + case Op_RoundVF: + case Op_RoundVD: + return vlen >= 8; + default: break; } @@ -140,16 +165,11 @@ source %{ } %} -definitions %{ - int_def VEC_COST (200, 200); -%} - // All VEC instructions // vector load/store instruct loadV(vReg dst, vmemA mem) %{ match(Set dst (LoadVector mem)); - ins_cost(VEC_COST); format %{ "loadV $dst, $mem\t# vector (rvv)" %} ins_encode %{ VectorRegister dst_reg = as_VectorRegister($dst$$reg); @@ -161,7 +181,6 @@ instruct loadV(vReg dst, vmemA mem) %{ instruct storeV(vReg src, vmemA mem) %{ match(Set mem (StoreVector mem src)); - ins_cost(VEC_COST); format %{ "storeV $mem, $src\t# vector (rvv)" %} ins_encode %{ VectorRegister src_reg = as_VectorRegister($src$$reg); @@ -249,7 +268,6 @@ instruct vmaskcmp_fp(vRegMask dst, vReg src1, vReg src2, immI cond) %{ predicate(Matcher::vector_element_basic_type(n) == T_FLOAT || Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst (VectorMaskCmp (Binary src1 src2) cond)); - effect(TEMP_DEF dst); format %{ "vmaskcmp_fp $dst, $src1, $src2, $cond" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -284,7 +302,6 @@ instruct vabs(vReg dst, vReg src, vReg tmp) %{ match(Set dst (AbsVS src)); match(Set dst (AbsVI src)); match(Set dst (AbsVL src)); - ins_cost(VEC_COST); effect(TEMP tmp); format %{ "vabs $dst, $src\t# KILL $tmp" %} ins_encode %{ @@ -299,7 +316,6 @@ instruct vabs(vReg dst, vReg src, vReg tmp) %{ instruct vabs_fp(vReg dst, vReg src) %{ match(Set dst (AbsVF src)); match(Set dst (AbsVD src)); - ins_cost(VEC_COST); format %{ "vabs_fp $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -316,7 +332,6 @@ instruct vabs_masked(vReg dst_src, vRegMask_V0 v0, vReg tmp) %{ match(Set dst_src (AbsVS dst_src v0)); match(Set dst_src (AbsVI dst_src v0)); match(Set dst_src (AbsVL dst_src v0)); - ins_cost(VEC_COST); effect(TEMP tmp); format %{ "vabs_masked $dst_src, $dst_src, $v0\t# KILL $tmp" %} ins_encode %{ @@ -333,7 +348,6 @@ instruct vabs_masked(vReg dst_src, vRegMask_V0 v0, vReg tmp) %{ instruct vabs_fp_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (AbsVF dst_src v0)); match(Set dst_src (AbsVD dst_src v0)); - ins_cost(VEC_COST); format %{ "vabs_fp_masked $dst_src, $dst_src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -350,7 +364,6 @@ instruct vadd(vReg dst, vReg src1, vReg src2) %{ match(Set dst (AddVS src1 src2)); match(Set dst (AddVI src1 src2)); match(Set dst (AddVL src1 src2)); - ins_cost(VEC_COST); format %{ "vadd $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -362,10 +375,23 @@ instruct vadd(vReg dst, vReg src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vadd_hfp(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (AddVHF src1 src2)); + format %{ "vadd_hfp $dst, $src1, $src2" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfadd_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + instruct vadd_fp(vReg dst, vReg src1, vReg src2) %{ match(Set dst (AddVF src1 src2)); match(Set dst (AddVD src1 src2)); - ins_cost(VEC_COST); format %{ "vadd_fp $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -384,7 +410,6 @@ instruct vadd_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (AddVS (Binary dst_src1 src2) v0)); match(Set dst_src1 (AddVI (Binary dst_src1 src2) v0)); match(Set dst_src1 (AddVL (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vadd_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -399,7 +424,6 @@ instruct vadd_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ instruct vadd_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (AddVF (Binary dst_src1 src2) v0)); match(Set dst_src1 (AddVD (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vadd_fp_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -413,11 +437,11 @@ instruct vadd_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ // vector-immediate add (unpredicated) -instruct vadd_immI(vReg dst, vReg src1, immI5 con) %{ +instruct vadd_vi(vReg dst, vReg src1, immI5 con) %{ match(Set dst (AddVB src1 (Replicate con))); match(Set dst (AddVS src1 (Replicate con))); match(Set dst (AddVI src1 (Replicate con))); - format %{ "vadd_immI $dst, $src1, $con" %} + format %{ "vadd_vi $dst, $src1, $con" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -428,9 +452,9 @@ instruct vadd_immI(vReg dst, vReg src1, immI5 con) %{ ins_pipe(pipe_slow); %} -instruct vadd_immL(vReg dst, vReg src1, immL5 con) %{ +instruct vaddL_vi(vReg dst, vReg src1, immL5 con) %{ match(Set dst (AddVL src1 (Replicate con))); - format %{ "vadd_immL $dst, $src1, $con" %} + format %{ "vaddL_vi $dst, $src1, $con" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vadd_vi(as_VectorRegister($dst$$reg), @@ -442,11 +466,11 @@ instruct vadd_immL(vReg dst, vReg src1, immL5 con) %{ // vector-scalar add (unpredicated) -instruct vadd_regI(vReg dst, vReg src1, iRegIorL2I src2) %{ +instruct vadd_vx(vReg dst, vReg src1, iRegIorL2I src2) %{ match(Set dst (AddVB src1 (Replicate src2))); match(Set dst (AddVS src1 (Replicate src2))); match(Set dst (AddVI src1 (Replicate src2))); - format %{ "vadd_regI $dst, $src1, $src2" %} + format %{ "vadd_vx $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -457,9 +481,9 @@ instruct vadd_regI(vReg dst, vReg src1, iRegIorL2I src2) %{ ins_pipe(pipe_slow); %} -instruct vadd_regL(vReg dst, vReg src1, iRegL src2) %{ +instruct vaddL_vx(vReg dst, vReg src1, iRegL src2) %{ match(Set dst (AddVL src1 (Replicate src2))); - format %{ "vadd_regL $dst, $src1, $src2" %} + format %{ "vaddL_vx $dst, $src1, $src2" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vadd_vx(as_VectorRegister($dst$$reg), @@ -471,11 +495,11 @@ instruct vadd_regL(vReg dst, vReg src1, iRegL src2) %{ // vector-immediate add (predicated) -instruct vadd_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ +instruct vadd_vi_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ match(Set dst_src (AddVB (Binary dst_src (Replicate con)) v0)); match(Set dst_src (AddVS (Binary dst_src (Replicate con)) v0)); match(Set dst_src (AddVI (Binary dst_src (Replicate con)) v0)); - format %{ "vadd_immI_masked $dst_src, $dst_src, $con" %} + format %{ "vadd_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -486,9 +510,9 @@ instruct vadd_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vadd_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ +instruct vaddL_vi_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ match(Set dst_src (AddVL (Binary dst_src (Replicate con)) v0)); - format %{ "vadd_immL_masked $dst_src, $dst_src, $con" %} + format %{ "vaddL_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vadd_vi(as_VectorRegister($dst_src$$reg), @@ -500,11 +524,11 @@ instruct vadd_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ // vector-scalar add (predicated) -instruct vadd_regI_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ +instruct vadd_vx_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ match(Set dst_src (AddVB (Binary dst_src (Replicate src2)) v0)); match(Set dst_src (AddVS (Binary dst_src (Replicate src2)) v0)); match(Set dst_src (AddVI (Binary dst_src (Replicate src2)) v0)); - format %{ "vadd_regI_masked $dst_src, $dst_src, $src2" %} + format %{ "vadd_vx_masked $dst_src, $dst_src, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -515,9 +539,9 @@ instruct vadd_regI_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vadd_regL_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ +instruct vaddL_vx_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ match(Set dst_src (AddVL (Binary dst_src (Replicate src2)) v0)); - format %{ "vadd_regL_masked $dst_src, $dst_src, $src2" %} + format %{ "vaddL_vx_masked $dst_src, $dst_src, $src2, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vadd_vx(as_VectorRegister($dst_src$$reg), @@ -534,7 +558,6 @@ instruct vsub(vReg dst, vReg src1, vReg src2) %{ match(Set dst (SubVS src1 src2)); match(Set dst (SubVI src1 src2)); match(Set dst (SubVL src1 src2)); - ins_cost(VEC_COST); format %{ "vsub $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -545,10 +568,22 @@ instruct vsub(vReg dst, vReg src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vsub_hfp(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (SubVHF src1 src2)); + format %{ "vsub_hfp $dst, $src1, $src2" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfsub_vv(as_VectorRegister($dst$$reg), as_VectorRegister($src1$$reg), + as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + instruct vsub_fp(vReg dst, vReg src1, vReg src2) %{ match(Set dst (SubVF src1 src2)); match(Set dst (SubVD src1 src2)); - ins_cost(VEC_COST); format %{ "vsub_fp $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -566,7 +601,6 @@ instruct vsub_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (SubVS (Binary dst_src1 src2) v0)); match(Set dst_src1 (SubVI (Binary dst_src1 src2) v0)); match(Set dst_src1 (SubVL (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vsub_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -580,7 +614,6 @@ instruct vsub_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ instruct vsub_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (SubVF (Binary dst_src1 src2) v0)); match(Set dst_src1 (SubVD (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vsub_fp_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -593,11 +626,11 @@ instruct vsub_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ // vector-scalar sub (unpredicated) -instruct vsub_regI(vReg dst, vReg src1, iRegIorL2I src2) %{ +instruct vsub_vx(vReg dst, vReg src1, iRegIorL2I src2) %{ match(Set dst (SubVB src1 (Replicate src2))); match(Set dst (SubVS src1 (Replicate src2))); match(Set dst (SubVI src1 (Replicate src2))); - format %{ "vsub_regI $dst, $src1, $src2" %} + format %{ "vsub_vx $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -608,9 +641,9 @@ instruct vsub_regI(vReg dst, vReg src1, iRegIorL2I src2) %{ ins_pipe(pipe_slow); %} -instruct vsub_regL(vReg dst, vReg src1, iRegL src2) %{ +instruct vsubL_vx(vReg dst, vReg src1, iRegL src2) %{ match(Set dst (SubVL src1 (Replicate src2))); - format %{ "vsub_regL $dst, $src1, $src2" %} + format %{ "vsubL_vx $dst, $src1, $src2" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vsub_vx(as_VectorRegister($dst$$reg), @@ -622,11 +655,11 @@ instruct vsub_regL(vReg dst, vReg src1, iRegL src2) %{ // vector-scalar sub (predicated) -instruct vsub_regI_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ +instruct vsub_vx_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ match(Set dst_src (SubVB (Binary dst_src (Replicate src2)) v0)); match(Set dst_src (SubVS (Binary dst_src (Replicate src2)) v0)); match(Set dst_src (SubVI (Binary dst_src (Replicate src2)) v0)); - format %{ "vsub_regI_masked $dst_src, $dst_src, $src2" %} + format %{ "vsub_vx_masked $dst_src, $dst_src, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -637,9 +670,9 @@ instruct vsub_regI_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vsub_regL_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ +instruct vsubL_vx_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ match(Set dst_src (SubVL (Binary dst_src (Replicate src2)) v0)); - format %{ "vsub_regL_masked $dst_src, $dst_src, $src2" %} + format %{ "vsubL_vx_masked $dst_src, $dst_src, $src2, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vsub_vx(as_VectorRegister($dst_src$$reg), @@ -649,11 +682,140 @@ instruct vsub_regL_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} +// -------- vector saturating integer operations + +// vector saturating signed integer addition + +instruct vsadd(vReg dst, vReg src1, vReg src2) %{ + predicate(n->is_SaturatingVector() && !n->as_SaturatingVector()->is_unsigned()); + match(Set dst (SaturatingAddV src1 src2)); + format %{ "vsadd $dst, $src1, $src2" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vsadd_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating unsigned integer addition + +instruct vsaddu(vReg dst, vReg src1, vReg src2) %{ + predicate(n->is_SaturatingVector() && n->as_SaturatingVector()->is_unsigned()); + match(Set dst (SaturatingAddV src1 src2)); + format %{ "vsaddu $dst, $src1, $src2" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vsaddu_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating signed integer addition (predicated) + +instruct vsadd_masked(vReg dst_src, vReg src1, vRegMask_V0 v0) %{ + predicate(n->is_SaturatingVector() && !n->as_SaturatingVector()->is_unsigned()); + match(Set dst_src (SaturatingAddV (Binary dst_src src1) v0)); + format %{ "vsadd_masked $dst_src, $dst_src, $src1, $v0" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vsadd_vv(as_VectorRegister($dst_src$$reg), as_VectorRegister($dst_src$$reg), + as_VectorRegister($src1$$reg), Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating unsigned integer addition (predicated) + +instruct vsaddu_masked(vReg dst_src, vReg src1, vRegMask_V0 v0) %{ + predicate(n->is_SaturatingVector() && n->as_SaturatingVector()->is_unsigned()); + match(Set dst_src (SaturatingAddV (Binary dst_src src1) v0)); + format %{ "vsaddu_masked $dst_src, $dst_src, $src1, $v0" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vsaddu_vv(as_VectorRegister($dst_src$$reg), as_VectorRegister($dst_src$$reg), + as_VectorRegister($src1$$reg), Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating signed integer subtraction + +instruct vssub(vReg dst, vReg src1, vReg src2) %{ + predicate(n->is_SaturatingVector() && !n->as_SaturatingVector()->is_unsigned()); + match(Set dst (SaturatingSubV src1 src2)); + format %{ "vssub $dst, $src1, $src2" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vssub_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating unsigned integer subtraction + +instruct vssubu(vReg dst, vReg src1, vReg src2) %{ + predicate(n->is_SaturatingVector() && n->as_SaturatingVector()->is_unsigned()); + match(Set dst (SaturatingSubV src1 src2)); + format %{ "vssubu $dst, $src1, $src2" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vssubu_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating signed integer subtraction (predicated) + +instruct vssub_masked(vReg dst_src, vReg src1, vRegMask_V0 v0) %{ + predicate(n->is_SaturatingVector() && !n->as_SaturatingVector()->is_unsigned()); + match(Set dst_src (SaturatingSubV (Binary dst_src src1) v0)); + format %{ "vssub_masked $dst_src, $dst_src, $src1, $v0" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vssub_vv(as_VectorRegister($dst_src$$reg), as_VectorRegister($dst_src$$reg), + as_VectorRegister($src1$$reg), Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +// vector saturating unsigned integer subtraction (predicated) + +instruct vssubu_masked(vReg dst_src, vReg src1, vRegMask_V0 v0) %{ + predicate(n->is_SaturatingVector() && n->as_SaturatingVector()->is_unsigned()); + match(Set dst_src (SaturatingSubV (Binary dst_src src1) v0)); + format %{ "vssubu_masked $dst_src, $dst_src, $src1, $v0" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vssubu_vv(as_VectorRegister($dst_src$$reg), as_VectorRegister($dst_src$$reg), + as_VectorRegister($src1$$reg), Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + // vector and instruct vand(vReg dst, vReg src1, vReg src2) %{ match(Set dst (AndV src1 src2)); - ins_cost(VEC_COST); format %{ "vand $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -669,7 +831,6 @@ instruct vand(vReg dst, vReg src1, vReg src2) %{ instruct vand_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (AndV (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vand_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -683,30 +844,30 @@ instruct vand_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ // vector-immediate and (unpredicated) -instruct vand_immI(vReg dst_src, immI5 con) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); - match(Set dst_src (AndV dst_src (Replicate con))); - format %{ "vand_immI $dst_src, $dst_src, $con" %} +instruct vand_vi(vReg dst, vReg src1, immI5 con) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (AndV src1 (Replicate con))); + format %{ "vand_vi $dst, $src1, $con" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); - __ vand_vi(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), + __ vand_vi(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), $con$$constant); %} ins_pipe(pipe_slow); %} -instruct vand_immL(vReg dst_src, immL5 con) %{ +instruct vandL_vi(vReg dst, vReg src1, immL5 con) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); - match(Set dst_src (AndV dst_src (Replicate con))); - format %{ "vand_immL $dst_src, $dst_src, $con" %} + match(Set dst (AndV src1 (Replicate con))); + format %{ "vandL_vi $dst, $src1, $con" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); - __ vand_vi(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), + __ vand_vi(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), $con$$constant); %} ins_pipe(pipe_slow); @@ -714,43 +875,43 @@ instruct vand_immL(vReg dst_src, immL5 con) %{ // vector-scalar and (unpredicated) -instruct vand_regI(vReg dst_src, iRegIorL2I src) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); - match(Set dst_src (AndV dst_src (Replicate src))); - format %{ "vand_regI $dst_src, $dst_src, $src" %} +instruct vand_vx(vReg dst, vReg src1, iRegIorL2I src2) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (AndV src1 (Replicate src2))); + format %{ "vand_vx $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); - __ vand_vx(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), - as_Register($src$$reg)); + __ vand_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); %} ins_pipe(pipe_slow); %} -instruct vand_regL(vReg dst_src, iRegL src) %{ +instruct vandL_vx(vReg dst, vReg src1, iRegL src2) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); - match(Set dst_src (AndV dst_src (Replicate src))); - format %{ "vand_regL $dst_src, $dst_src, $src" %} + match(Set dst (AndV src1 (Replicate src2))); + format %{ "vandL_vx $dst, $src1, $src2" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); - __ vand_vx(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), - as_Register($src$$reg)); + __ vand_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); %} ins_pipe(pipe_slow); %} // vector-immediate and (predicated) -instruct vand_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vand_vi_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (AndV (Binary dst_src (Replicate con)) v0)); - format %{ "vand_immI_masked $dst_src, $dst_src, $con" %} + format %{ "vand_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -761,10 +922,10 @@ instruct vand_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vand_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ +instruct vandL_vi_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src (AndV (Binary dst_src (Replicate con)) v0)); - format %{ "vand_immL_masked $dst_src, $dst_src, $con" %} + format %{ "vandL_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vand_vi(as_VectorRegister($dst_src$$reg), @@ -776,12 +937,12 @@ instruct vand_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ // vector-scalar and (predicated) -instruct vand_regI_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vand_vx_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (AndV (Binary dst_src (Replicate src)) v0)); - format %{ "vand_regI_masked $dst_src, $dst_src, $src" %} + format %{ "vand_vx_masked $dst_src, $dst_src, $src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -792,10 +953,10 @@ instruct vand_regI_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vand_regL_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ +instruct vandL_vx_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src (AndV (Binary dst_src (Replicate src)) v0)); - format %{ "vand_regL_masked $dst_src, $dst_src, $src" %} + format %{ "vandL_vx_masked $dst_src, $dst_src, $src, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vand_vx(as_VectorRegister($dst_src$$reg), @@ -809,7 +970,6 @@ instruct vand_regL_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ instruct vor(vReg dst, vReg src1, vReg src2) %{ match(Set dst (OrV src1 src2)); - ins_cost(VEC_COST); format %{ "vor $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -825,7 +985,6 @@ instruct vor(vReg dst, vReg src1, vReg src2) %{ instruct vor_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (OrV (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vor_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -839,30 +998,30 @@ instruct vor_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ // vector-immediate or (unpredicated) -instruct vor_immI(vReg dst_src, immI5 con) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); - match(Set dst_src (OrV dst_src (Replicate con))); - format %{ "vor_immI $dst_src, $dst_src, $con" %} +instruct vor_vi(vReg dst, vReg src1, immI5 con) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (OrV src1 (Replicate con))); + format %{ "vor_vi $dst, $src1, $con" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); - __ vor_vi(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), + __ vor_vi(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), $con$$constant); %} ins_pipe(pipe_slow); %} -instruct vor_immL(vReg dst_src, immL5 con) %{ +instruct vorL_vi(vReg dst, vReg src1, immL5 con) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); - match(Set dst_src (OrV dst_src (Replicate con))); - format %{ "vor_immL $dst_src, $dst_src, $con" %} + match(Set dst (OrV src1 (Replicate con))); + format %{ "vorL_vi $dst, $src1, $con" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); - __ vor_vi(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), + __ vor_vi(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), $con$$constant); %} ins_pipe(pipe_slow); @@ -870,43 +1029,43 @@ instruct vor_immL(vReg dst_src, immL5 con) %{ // vector-scalar or (unpredicated) -instruct vor_regI(vReg dst_src, iRegIorL2I src) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); - match(Set dst_src (OrV dst_src (Replicate src))); - format %{ "vor_regI $dst_src, $dst_src, $src" %} +instruct vor_vx(vReg dst, vReg src1, iRegIorL2I src2) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (OrV src1 (Replicate src2))); + format %{ "vor_vx $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); - __ vor_vx(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), - as_Register($src$$reg)); + __ vor_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); %} ins_pipe(pipe_slow); %} -instruct vor_regL(vReg dst_src, iRegL src) %{ +instruct vorL_vx(vReg dst, vReg src1, iRegL src2) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); - match(Set dst_src (OrV dst_src (Replicate src))); - format %{ "vor_regL $dst_src, $dst_src, $src" %} + match(Set dst (OrV src1 (Replicate src2))); + format %{ "vorL_vx $dst, $src1, $src2" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); - __ vor_vx(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), - as_Register($src$$reg)); + __ vor_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); %} ins_pipe(pipe_slow); %} // vector-immediate or (predicated) -instruct vor_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vor_vi_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (OrV (Binary dst_src (Replicate con)) v0)); - format %{ "vor_immI_masked $dst_src, $dst_src, $con" %} + format %{ "vor_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -917,10 +1076,10 @@ instruct vor_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vor_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ +instruct vorL_vi_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src (OrV (Binary dst_src (Replicate con)) v0)); - format %{ "vor_immL_masked $dst_src, $dst_src, $con" %} + format %{ "vorL_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vor_vi(as_VectorRegister($dst_src$$reg), @@ -932,12 +1091,12 @@ instruct vor_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ // vector-scalar or (predicated) -instruct vor_regI_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vor_vx_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (OrV (Binary dst_src (Replicate src)) v0)); - format %{ "vor_regI_masked $dst_src, $dst_src, $src" %} + format %{ "vor_vx_masked $dst_src, $dst_src, $src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -948,10 +1107,10 @@ instruct vor_regI_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vor_regL_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ +instruct vorL_vx_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src (OrV (Binary dst_src (Replicate src)) v0)); - format %{ "vor_regL_masked $dst_src, $dst_src, $src" %} + format %{ "vorL_vx_masked $dst_src, $dst_src, $src, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vor_vx(as_VectorRegister($dst_src$$reg), @@ -965,7 +1124,6 @@ instruct vor_regL_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ instruct vxor(vReg dst, vReg src1, vReg src2) %{ match(Set dst (XorV src1 src2)); - ins_cost(VEC_COST); format %{ "vxor $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -981,7 +1139,6 @@ instruct vxor(vReg dst, vReg src1, vReg src2) %{ instruct vxor_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (XorV (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vxor_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -995,30 +1152,30 @@ instruct vxor_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ // vector-immediate xor (unpredicated) -instruct vxor_immI(vReg dst_src, immI5 con) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); - match(Set dst_src (XorV dst_src (Replicate con))); - format %{ "vxor_immI $dst_src, $dst_src, $con" %} +instruct vxor_vi(vReg dst, vReg src1, immI5 con) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (XorV src1 (Replicate con))); + format %{ "vxor_vi $dst, $src1, $con" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); - __ vxor_vi(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), + __ vxor_vi(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), $con$$constant); %} ins_pipe(pipe_slow); %} -instruct vxor_immL(vReg dst_src, immL5 con) %{ +instruct vxorL_vi(vReg dst, vReg src1, immL5 con) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); - match(Set dst_src (XorV dst_src (Replicate con))); - format %{ "vxor_immL $dst_src, $dst_src, $con" %} + match(Set dst (XorV src1 (Replicate con))); + format %{ "vxorL_vi $dst, $src1, $con" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); - __ vxor_vi(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), + __ vxor_vi(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), $con$$constant); %} ins_pipe(pipe_slow); @@ -1026,43 +1183,43 @@ instruct vxor_immL(vReg dst_src, immL5 con) %{ // vector-scalar xor (unpredicated) -instruct vxor_regI(vReg dst_src, iRegIorL2I src) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); - match(Set dst_src (XorV dst_src (Replicate src))); - format %{ "vxor_regI $dst_src, $dst_src, $src" %} +instruct vxor_vx(vReg dst, vReg src1, iRegIorL2I src2) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (XorV src1 (Replicate src2))); + format %{ "vxor_vx $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); - __ vxor_vx(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), - as_Register($src$$reg)); + __ vxor_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); %} ins_pipe(pipe_slow); %} -instruct vxor_regL(vReg dst_src, iRegL src) %{ +instruct vxorL_vx(vReg dst, vReg src1, iRegL src2) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); - match(Set dst_src (XorV dst_src (Replicate src))); - format %{ "vxor_regL $dst_src, $dst_src, $src" %} + match(Set dst (XorV src1 (Replicate src2))); + format %{ "vxorL_vx $dst, $src1, $src2" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); - __ vxor_vx(as_VectorRegister($dst_src$$reg), - as_VectorRegister($dst_src$$reg), - as_Register($src$$reg)); + __ vxor_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); %} ins_pipe(pipe_slow); %} // vector-immediate xor (predicated) -instruct vxor_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vxor_vi_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (XorV (Binary dst_src (Replicate con)) v0)); - format %{ "vxor_immI_masked $dst_src, $dst_src, $con" %} + format %{ "vxor_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -1073,10 +1230,10 @@ instruct vxor_immI_masked(vReg dst_src, immI5 con, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vxor_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ +instruct vxorL_vi_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src (XorV (Binary dst_src (Replicate con)) v0)); - format %{ "vxor_immL_masked $dst_src, $dst_src, $con" %} + format %{ "vxorL_vi_masked $dst_src, $dst_src, $con, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vxor_vi(as_VectorRegister($dst_src$$reg), @@ -1088,12 +1245,12 @@ instruct vxor_immL_masked(vReg dst_src, immL5 con, vRegMask_V0 v0) %{ // vector-scalar xor (predicated) -instruct vxor_regI_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vxor_vx_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (XorV (Binary dst_src (Replicate src)) v0)); - format %{ "vxor_regI_masked $dst_src, $dst_src, $src" %} + format %{ "vxor_vx_masked $dst_src, $dst_src, $src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -1104,10 +1261,10 @@ instruct vxor_regI_masked(vReg dst_src, iRegIorL2I src, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vxor_regL_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ +instruct vxorL_vx_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src (XorV (Binary dst_src (Replicate src)) v0)); - format %{ "vxor_regL_masked $dst_src, $dst_src, $src" %} + format %{ "vxorL_vx_masked $dst_src, $dst_src, $src, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vxor_vx(as_VectorRegister($dst_src$$reg), @@ -1121,16 +1278,38 @@ instruct vxor_regL_masked(vReg dst_src, iRegL src, vRegMask_V0 v0) %{ // vector and not +instruct vand_notB(vReg dst, vReg src1, vReg src2, immI_M1 m1) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_BYTE); + match(Set dst (AndV src1 (XorV src2 (Replicate m1)))); + format %{ "vand_notB $dst, $src1, $src2" %} + ins_encode %{ + __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); + __ vandn_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notS(vReg dst, vReg src1, vReg src2, immI_M1 m1) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst (AndV src1 (XorV src2 (Replicate m1)))); + format %{ "vand_notS $dst, $src1, $src2" %} + ins_encode %{ + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vandn_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + instruct vand_notI(vReg dst, vReg src1, vReg src2, immI_M1 m1) %{ - predicate(UseZvbb); - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_INT); match(Set dst (AndV src1 (XorV src2 (Replicate m1)))); format %{ "vand_notI $dst, $src1, $src2" %} ins_encode %{ - BasicType bt = Matcher::vector_element_basic_type(this); - __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vsetvli_helper(T_INT, Matcher::vector_length(this)); __ vandn_vv(as_VectorRegister($dst$$reg), as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); @@ -1139,8 +1318,7 @@ instruct vand_notI(vReg dst, vReg src1, vReg src2, immI_M1 m1) %{ %} instruct vand_notL(vReg dst, vReg src1, vReg src2, immL_M1 m1) %{ - predicate(UseZvbb); - predicate(Matcher::vector_element_basic_type(n) == T_LONG); + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst (AndV src1 (XorV src2 (Replicate m1)))); format %{ "vand_notL $dst, $src1, $src2" %} ins_encode %{ @@ -1152,16 +1330,40 @@ instruct vand_notL(vReg dst, vReg src1, vReg src2, immL_M1 m1) %{ ins_pipe(pipe_slow); %} +instruct vand_notB_masked(vReg dst_src1, vReg src2, immI_M1 m1, vRegMask_V0 v0) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_BYTE); + match(Set dst_src1 (AndV (Binary dst_src1 (XorV src2 (Replicate m1))) v0)); + format %{ "vand_notB_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); + __ vandn_vv(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($dst_src1$$reg), + as_VectorRegister($src2$$reg), + Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notS_masked(vReg dst_src1, vReg src2, immI_M1 m1, vRegMask_V0 v0) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst_src1 (AndV (Binary dst_src1 (XorV src2 (Replicate m1))) v0)); + format %{ "vand_notS_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vandn_vv(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($dst_src1$$reg), + as_VectorRegister($src2$$reg), + Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + instruct vand_notI_masked(vReg dst_src1, vReg src2, immI_M1 m1, vRegMask_V0 v0) %{ - predicate(UseZvbb); - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src1 (AndV (Binary dst_src1 (XorV src2 (Replicate m1))) v0)); format %{ "vand_notI_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ - BasicType bt = Matcher::vector_element_basic_type(this); - __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vsetvli_helper(T_INT, Matcher::vector_length(this)); __ vandn_vv(as_VectorRegister($dst_src1$$reg), as_VectorRegister($dst_src1$$reg), as_VectorRegister($src2$$reg), @@ -1171,8 +1373,7 @@ instruct vand_notI_masked(vReg dst_src1, vReg src2, immI_M1 m1, vRegMask_V0 v0) %} instruct vand_notL_masked(vReg dst_src1, vReg src2, immL_M1 m1, vRegMask_V0 v0) %{ - predicate(UseZvbb); - predicate(Matcher::vector_element_basic_type(n) == T_LONG); + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst_src1 (AndV (Binary dst_src1 (XorV src2 (Replicate m1))) v0)); format %{ "vand_notL_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ @@ -1185,16 +1386,124 @@ instruct vand_notL_masked(vReg dst_src1, vReg src2, immL_M1 m1, vRegMask_V0 v0) ins_pipe(pipe_slow); %} +instruct vand_notB_vx(vReg dst, vReg src1, iRegIorL2I src2, immI_M1 m1) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_BYTE); + match(Set dst (AndV src1 (Replicate (XorI src2 m1)))); + format %{ "vand_notB_vx $dst, $src1, $src2" %} + ins_encode %{ + __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notS_vx(vReg dst, vReg src1, iRegIorL2I src2, immI_M1 m1) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst (AndV src1 (Replicate (XorI src2 m1)))); + format %{ "vand_notS_vx $dst, $src1, $src2" %} + ins_encode %{ + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notI_vx(vReg dst, vReg src1, iRegIorL2I src2, immI_M1 m1) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst (AndV src1 (Replicate (XorI src2 m1)))); + format %{ "vand_notI_vx $dst, $src1, $src2" %} + ins_encode %{ + __ vsetvli_helper(T_INT, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notL_vx(vReg dst, vReg src1, iRegL src2, immL_M1 m1) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_LONG); + match(Set dst (AndV src1 (Replicate (XorL src2 m1)))); + format %{ "vand_notL_vx $dst, $src1, $src2" %} + ins_encode %{ + __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_Register($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notB_vx_masked(vReg dst_src1, iRegIorL2I src2, immI_M1 m1, vRegMask_V0 v0) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_BYTE); + match(Set dst_src1 (AndV (Binary dst_src1 (Replicate (XorI src2 m1))) v0)); + format %{ "vand_notB_vx_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($dst_src1$$reg), + as_Register($src2$$reg), + Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notS_vx_masked(vReg dst_src1, iRegIorL2I src2, immI_M1 m1, vRegMask_V0 v0) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst_src1 (AndV (Binary dst_src1 (Replicate (XorI src2 m1))) v0)); + format %{ "vand_notS_vx_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($dst_src1$$reg), + as_Register($src2$$reg), + Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notI_vx_masked(vReg dst_src1, iRegIorL2I src2, immI_M1 m1, vRegMask_V0 v0) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_INT); + match(Set dst_src1 (AndV (Binary dst_src1 (Replicate (XorI src2 m1))) v0)); + format %{ "vand_notI_vx_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + __ vsetvli_helper(T_INT, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($dst_src1$$reg), + as_Register($src2$$reg), + Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +instruct vand_notL_vx_masked(vReg dst_src1, iRegL src2, immL_M1 m1, vRegMask_V0 v0) %{ + predicate(UseZvbb && Matcher::vector_element_basic_type(n) == T_LONG); + match(Set dst_src1 (AndV (Binary dst_src1 (Replicate (XorL src2 m1))) v0)); + format %{ "vand_notL_vx_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); + __ vandn_vx(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($dst_src1$$reg), + as_Register($src2$$reg), + Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + // ------------------------------ Vector not ----------------------------------- // vector not -instruct vnotI(vReg dst, vReg src, immI_M1 m1) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vnot(vReg dst, vReg src, immI_M1 m1) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst (XorV src (Replicate m1))); - format %{ "vnotI $dst, $src" %} + format %{ "vnot $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -1220,12 +1529,12 @@ instruct vnotL(vReg dst, vReg src, immL_M1 m1) %{ // vector not - predicated -instruct vnotI_masked(vReg dst_src, immI_M1 m1, vRegMask_V0 v0) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); +instruct vnot_masked(vReg dst_src, immI_M1 m1, vRegMask_V0 v0) %{ + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst_src (XorV (Binary dst_src (Replicate m1)) v0)); - format %{ "vnotI_masked $dst_src, $dst_src, $v0" %} + format %{ "vnot_masked $dst_src, $dst_src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -1251,10 +1560,23 @@ instruct vnotL_masked(vReg dst_src, immI_M1 m1, vRegMask_V0 v0) %{ // vector float div +instruct vdiv_hfp(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (DivVHF src1 src2)); + format %{ "vdiv_hfp $dst, $src1, $src2" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfdiv_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), + as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + instruct vdiv_fp(vReg dst, vReg src1, vReg src2) %{ match(Set dst (DivVF src1 src2)); match(Set dst (DivVD src1 src2)); - ins_cost(VEC_COST); format %{ "vdiv_fp $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1271,7 +1593,6 @@ instruct vdiv_fp(vReg dst, vReg src1, vReg src2) %{ instruct vdiv_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (DivVF (Binary dst_src1 src2) v0)); match(Set dst_src1 (DivVD (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vdiv_fp_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1289,7 +1610,6 @@ instruct vmax(vReg dst, vReg src1, vReg src2) %{ predicate(Matcher::vector_element_basic_type(n) != T_FLOAT && Matcher::vector_element_basic_type(n) != T_DOUBLE); match(Set dst (MaxV src1 src2)); - ins_cost(VEC_COST); format %{ "vmax $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1304,7 +1624,6 @@ instruct vmin(vReg dst, vReg src1, vReg src2) %{ predicate(Matcher::vector_element_basic_type(n) != T_FLOAT && Matcher::vector_element_basic_type(n) != T_DOUBLE); match(Set dst (MinV src1 src2)); - ins_cost(VEC_COST); format %{ "vmin $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1321,7 +1640,6 @@ instruct vmax_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) != T_FLOAT && Matcher::vector_element_basic_type(n) != T_DOUBLE); match(Set dst_src1 (MaxV (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vmax_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1336,7 +1654,6 @@ instruct vmin_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ predicate(Matcher::vector_element_basic_type(n) != T_FLOAT && Matcher::vector_element_basic_type(n) != T_DOUBLE); match(Set dst_src1 (MinV (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vmin_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1347,6 +1664,92 @@ instruct vmin_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} +// vector unsigned integer max/min + +instruct vmaxu(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (UMaxV src1 src2)); + format %{ "vmaxu $dst, $src1, $src2" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vmaxu_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +instruct vminu(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (UMinV src1 src2)); + format %{ "vminu $dst, $src1, $src2" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vminu_vv(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + +// vector unsigned integer max/min - predicated + +instruct vmaxu_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ + match(Set dst_src1 (UMaxV (Binary dst_src1 src2) v0)); + format %{ "vmaxu_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vmaxu_vv(as_VectorRegister($dst_src1$$reg), as_VectorRegister($dst_src1$$reg), + as_VectorRegister($src2$$reg), Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +instruct vminu_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ + match(Set dst_src1 (UMinV (Binary dst_src1 src2) v0)); + format %{ "vminu_masked $dst_src1, $dst_src1, $src2, $v0" %} + ins_encode %{ + BasicType bt = Matcher::vector_element_basic_type(this); + assert(is_integral_type(bt), "unsupported type"); + __ vsetvli_helper(bt, Matcher::vector_length(this)); + __ vminu_vv(as_VectorRegister($dst_src1$$reg), as_VectorRegister($dst_src1$$reg), + as_VectorRegister($src2$$reg), Assembler::v0_t); + %} + ins_pipe(pipe_slow); +%} + +// vector float-point max/min (half precision) + +instruct vmax_hfp(vReg dst, vReg src1, vReg src2, vRegMask_V0 v0) %{ + match(Set dst (MaxVHF src1 src2)); + effect(TEMP_DEF dst, TEMP v0); + format %{ "vmax_hfp $dst, $src1, $src2" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ minmax_fp_v(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg), + T_SHORT, false /* is_min */, Matcher::vector_length(this)); + %} + ins_pipe(pipe_slow); +%} + +instruct vmin_hfp(vReg dst, vReg src1, vReg src2, vRegMask_V0 v0) %{ + match(Set dst (MinVHF src1 src2)); + effect(TEMP_DEF dst, TEMP v0); + format %{ "vmin_hfp $dst, $src1, $src2" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ minmax_fp_v(as_VectorRegister($dst$$reg), + as_VectorRegister($src1$$reg), as_VectorRegister($src2$$reg), + T_SHORT, true /* is_min */, Matcher::vector_length(this)); + %} + ins_pipe(pipe_slow); +%} + // vector float-point max/min instruct vmax_fp(vReg dst, vReg src1, vReg src2, vRegMask_V0 v0) %{ @@ -1354,7 +1757,6 @@ instruct vmax_fp(vReg dst, vReg src1, vReg src2, vRegMask_V0 v0) %{ Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst (MaxV src1 src2)); effect(TEMP_DEF dst, TEMP v0); - ins_cost(VEC_COST); format %{ "vmax_fp $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1370,7 +1772,6 @@ instruct vmin_fp(vReg dst, vReg src1, vReg src2, vRegMask_V0 v0) %{ Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst (MinV src1 src2)); effect(TEMP_DEF dst, TEMP v0); - ins_cost(VEC_COST); format %{ "vmin_fp $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1388,7 +1789,6 @@ instruct vmax_fp_masked(vReg dst_src1, vReg src2, vRegMask vmask, vReg tmp1, vRe Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst_src1 (MaxV (Binary dst_src1 src2) vmask)); effect(TEMP_DEF dst_src1, TEMP tmp1, TEMP tmp2, TEMP v0); - ins_cost(VEC_COST); format %{ "vmax_fp_masked $dst_src1, $dst_src1, $src2, $vmask\t# KILL $tmp1, $tmp2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1405,7 +1805,6 @@ instruct vmin_fp_masked(vReg dst_src1, vReg src2, vRegMask vmask, vReg tmp1, vRe Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst_src1 (MinV (Binary dst_src1 src2) vmask)); effect(TEMP_DEF dst_src1, TEMP tmp1, TEMP tmp2, TEMP v0); - ins_cost(VEC_COST); format %{ "vmin_fp_masked $dst_src1, $dst_src1, $src2, $vmask\t# KILL $tmp1, $tmp2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1419,11 +1818,25 @@ instruct vmin_fp_masked(vReg dst_src1, vReg src2, vRegMask vmask, vReg tmp1, vRe // vector fmla +// dst_src1 = src2 * src3 + dst_src1 (half precision) +instruct vhfmla(vReg dst_src1, vReg src2, vReg src3) %{ + match(Set dst_src1 (FmaVHF dst_src1 (Binary src2 src3))); + format %{ "vhfmla $dst_src1, $dst_src1, $src2, $src3" %} + ins_encode %{ + assert(UseFMA, "Needs FMA instructions support."); + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfmacc_vv(as_VectorRegister($dst_src1$$reg), + as_VectorRegister($src2$$reg), as_VectorRegister($src3$$reg)); + %} + ins_pipe(pipe_slow); +%} + // dst_src1 = src2 * src3 + dst_src1 instruct vfmla(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVF dst_src1 (Binary src2 src3))); match(Set dst_src1 (FmaVD dst_src1 (Binary src2 src3))); - ins_cost(VEC_COST); format %{ "vfmla $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1458,7 +1871,6 @@ instruct vfmadd_masked(vReg dst_src1, vReg src2, vReg src3, vRegMask_V0 v0) %{ // "(-src2) * src3 + dst_src1" has been idealized to "src3 * (-src2) + dst_src1" instruct vfmlsF(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVF dst_src1 (Binary src2 (NegVF src3)))); - ins_cost(VEC_COST); format %{ "vfmlsF $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1473,7 +1885,6 @@ instruct vfmlsF(vReg dst_src1, vReg src2, vReg src3) %{ // "(-src2) * src3 + dst_src1" has been idealized to "src3 * (-src2) + dst_src1" instruct vfmlsD(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVD dst_src1 (Binary src2 (NegVD src3)))); - ins_cost(VEC_COST); format %{ "vfmlsD $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1507,7 +1918,6 @@ instruct vfnmsub_masked(vReg dst_src1, vReg src2, vReg src3, vRegMask_V0 v0) %{ // "(-src2) * src3 - dst_src1" has been idealized to "src3 * (-src2) - dst_src1" instruct vfnmlaF(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVF (NegVF dst_src1) (Binary src2 (NegVF src3)))); - ins_cost(VEC_COST); format %{ "vfnmlaF $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1522,7 +1932,6 @@ instruct vfnmlaF(vReg dst_src1, vReg src2, vReg src3) %{ // "(-src2) * src3 - dst_src1" has been idealized to "src3 * (-src2) - dst_src1" instruct vfnmlaD(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVD (NegVD dst_src1) (Binary src2 (NegVD src3)))); - ins_cost(VEC_COST); format %{ "vfnmlaD $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1555,7 +1964,6 @@ instruct vfnmadd_masked(vReg dst_src1, vReg src2, vReg src3, vRegMask_V0 v0) %{ // dst_src1 = src2 * src3 - dst_src1 instruct vfnmlsF(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVF (NegVF dst_src1) (Binary src2 src3))); - ins_cost(VEC_COST); format %{ "vfnmlsF $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1569,7 +1977,6 @@ instruct vfnmlsF(vReg dst_src1, vReg src2, vReg src3) %{ // dst_src1 = -dst_src1 + src2 * src3 instruct vfnmlsD(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (FmaVD (NegVD dst_src1) (Binary src2 src3))); - ins_cost(VEC_COST); format %{ "vfnmlsD $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ assert(UseFMA, "Needs FMA instructions support."); @@ -1605,7 +2012,6 @@ instruct vmla(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (AddVS dst_src1 (MulVS src2 src3))); match(Set dst_src1 (AddVI dst_src1 (MulVI src2 src3))); match(Set dst_src1 (AddVL dst_src1 (MulVL src2 src3))); - ins_cost(VEC_COST); format %{ "vmla $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1641,7 +2047,6 @@ instruct vmls(vReg dst_src1, vReg src2, vReg src3) %{ match(Set dst_src1 (SubVS dst_src1 (MulVS src2 src3))); match(Set dst_src1 (SubVI dst_src1 (MulVI src2 src3))); match(Set dst_src1 (SubVL dst_src1 (MulVL src2 src3))); - ins_cost(VEC_COST); format %{ "vmls $dst_src1, $dst_src1, $src2, $src3" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1676,7 +2081,6 @@ instruct vmul(vReg dst, vReg src1, vReg src2) %{ match(Set dst (MulVS src1 src2)); match(Set dst (MulVI src1 src2)); match(Set dst (MulVL src1 src2)); - ins_cost(VEC_COST); format %{ "vmul $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1687,10 +2091,22 @@ instruct vmul(vReg dst, vReg src1, vReg src2) %{ ins_pipe(pipe_slow); %} +instruct vmul_hfp(vReg dst, vReg src1, vReg src2) %{ + match(Set dst (MulVHF src1 src2)); + format %{ "vmul_hfp $dst, $src1, $src2" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfmul_vv(as_VectorRegister($dst$$reg), as_VectorRegister($src1$$reg), + as_VectorRegister($src2$$reg)); + %} + ins_pipe(pipe_slow); +%} + instruct vmul_fp(vReg dst, vReg src1, vReg src2) %{ match(Set dst (MulVF src1 src2)); match(Set dst (MulVD src1 src2)); - ins_cost(VEC_COST); format %{ "vmul_fp $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1708,7 +2124,6 @@ instruct vmul_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (MulVS (Binary dst_src1 src2) v0)); match(Set dst_src1 (MulVI (Binary dst_src1 src2) v0)); match(Set dst_src1 (MulVL (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vmul_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1722,7 +2137,6 @@ instruct vmul_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ instruct vmul_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ match(Set dst_src1 (MulVF (Binary dst_src1 src2) v0)); match(Set dst_src1 (MulVD (Binary dst_src1 src2) v0)); - ins_cost(VEC_COST); format %{ "vmul_fp_masked $dst_src1, $dst_src1, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1735,11 +2149,11 @@ instruct vmul_fp_masked(vReg dst_src1, vReg src2, vRegMask_V0 v0) %{ // vector-scalar mul (unpredicated) -instruct vmul_regI(vReg dst, vReg src1, iRegIorL2I src2) %{ +instruct vmul_vx(vReg dst, vReg src1, iRegIorL2I src2) %{ match(Set dst (MulVB src1 (Replicate src2))); match(Set dst (MulVS src1 (Replicate src2))); match(Set dst (MulVI src1 (Replicate src2))); - format %{ "vmul_regI $dst, $src1, $src2" %} + format %{ "vmul_vx $dst, $src1, $src2" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -1750,9 +2164,9 @@ instruct vmul_regI(vReg dst, vReg src1, iRegIorL2I src2) %{ ins_pipe(pipe_slow); %} -instruct vmul_regL(vReg dst, vReg src1, iRegL src2) %{ +instruct vmulL_vx(vReg dst, vReg src1, iRegL src2) %{ match(Set dst (MulVL src1 (Replicate src2))); - format %{ "vmul_regL $dst, $src1, $src2" %} + format %{ "vmulL_vx $dst, $src1, $src2" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vmul_vx(as_VectorRegister($dst$$reg), @@ -1764,11 +2178,11 @@ instruct vmul_regL(vReg dst, vReg src1, iRegL src2) %{ // vector-scalar mul (predicated) -instruct vmul_regI_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ +instruct vmul_vx_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ match(Set dst_src (MulVB (Binary dst_src (Replicate src2)) v0)); match(Set dst_src (MulVS (Binary dst_src (Replicate src2)) v0)); match(Set dst_src (MulVI (Binary dst_src (Replicate src2)) v0)); - format %{ "vmul_regI_masked $dst_src, $dst_src, $src2" %} + format %{ "vmul_vx_masked $dst_src, $dst_src, $src2, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -1779,9 +2193,9 @@ instruct vmul_regI_masked(vReg dst_src, iRegIorL2I src2, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vmul_regL_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ +instruct vmulL_vx_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ match(Set dst_src (MulVL (Binary dst_src (Replicate src2)) v0)); - format %{ "vmul_regL_masked $dst_src, $dst_src, $src2" %} + format %{ "vmulL_vx_masked $dst_src, $dst_src, $src2, $v0" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); __ vmul_vx(as_VectorRegister($dst_src$$reg), @@ -1796,7 +2210,6 @@ instruct vmul_regL_masked(vReg dst_src, iRegL src2, vRegMask_V0 v0) %{ instruct vneg(vReg dst, vReg src) %{ match(Set dst (NegVI src)); match(Set dst (NegVL src)); - ins_cost(VEC_COST); format %{ "vneg $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1811,7 +2224,6 @@ instruct vneg(vReg dst, vReg src) %{ instruct vneg_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (NegVI dst_src v0)); match(Set dst_src (NegVL dst_src v0)); - ins_cost(VEC_COST); format %{ "vneg_masked $dst_src, $dst_src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1827,7 +2239,6 @@ instruct vneg_masked(vReg dst_src, vRegMask_V0 v0) %{ instruct vfneg(vReg dst, vReg src) %{ match(Set dst (NegVF src)); match(Set dst (NegVD src)); - ins_cost(VEC_COST); format %{ "vfneg $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1842,7 +2253,6 @@ instruct vfneg(vReg dst, vReg src) %{ instruct vfneg_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (NegVF dst_src v0)); match(Set dst_src (NegVD dst_src v0)); - ins_cost(VEC_COST); format %{ "vfneg_masked $dst_src, $dst_src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -1855,14 +2265,13 @@ instruct vfneg_masked(vReg dst_src, vRegMask_V0 v0) %{ // vector and reduction -instruct reduce_andI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ +instruct reduce_and(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (AndReductionV src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_andI $dst, $src1, $src2\t# KILL $tmp" %} + format %{ "reduce_and $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -1876,7 +2285,6 @@ instruct reduce_andL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (AndReductionV src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_andL $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -1889,14 +2297,13 @@ instruct reduce_andL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ // vector and reduction - predicated -instruct reduce_andI_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ +instruct reduce_and_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (AndReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_andI_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} + format %{ "reduce_and_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -1911,7 +2318,6 @@ instruct reduce_andL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0 predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (AndReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_andL_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -1925,14 +2331,13 @@ instruct reduce_andL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0 // vector or reduction -instruct reduce_orI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ +instruct reduce_or(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (OrReductionV src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_orI $dst, $src1, $src2\t# KILL $tmp" %} + format %{ "reduce_or $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -1946,7 +2351,6 @@ instruct reduce_orL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (OrReductionV src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_orL $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -1959,14 +2363,13 @@ instruct reduce_orL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ // vector or reduction - predicated -instruct reduce_orI_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ +instruct reduce_or_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (OrReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_orI_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} + format %{ "reduce_or_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -1981,7 +2384,6 @@ instruct reduce_orL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0, predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (OrReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_orL_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -1995,14 +2397,13 @@ instruct reduce_orL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0, // vector xor reduction -instruct reduce_xorI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ +instruct reduce_xor(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (XorReductionV src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_xorI $dst, $src1, $src2\t# KILL $tmp" %} + format %{ "reduce_xor $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2016,7 +2417,6 @@ instruct reduce_xorL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (XorReductionV src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_xorL $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -2029,14 +2429,13 @@ instruct reduce_xorL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ // vector xor reduction - predicated -instruct reduce_xorI_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ +instruct reduce_xor_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (XorReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_xorI_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} + format %{ "reduce_xor_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2051,7 +2450,6 @@ instruct reduce_xorL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0 predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (XorReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_xorL_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -2065,14 +2463,13 @@ instruct reduce_xorL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0 // vector add reduction -instruct reduce_addI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ +instruct reduce_add(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (AddReductionVI src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_addI $dst, $src1, $src2\t# KILL $tmp" %} + format %{ "reduce_add $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2086,7 +2483,6 @@ instruct reduce_addL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (AddReductionVL src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addL $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -2109,7 +2505,6 @@ instruct reduce_addF_ordered(fRegF dst, fRegF src1, vReg src2, vReg tmp) %{ predicate(n->as_Reduction()->requires_strict_order()); match(Set dst (AddReductionVF src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addF_ordered $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ __ vsetvli_helper(T_FLOAT, Matcher::vector_length(this, $src2)); @@ -2125,7 +2520,6 @@ instruct reduce_addF_unordered(fRegF dst, fRegF src1, vReg src2, vReg tmp) %{ predicate(!n->as_Reduction()->requires_strict_order()); match(Set dst (AddReductionVF src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addF_unordered $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ __ vsetvli_helper(T_FLOAT, Matcher::vector_length(this, $src2)); @@ -2141,7 +2535,6 @@ instruct reduce_addD_ordered(fRegD dst, fRegD src1, vReg src2, vReg tmp) %{ predicate(n->as_Reduction()->requires_strict_order()); match(Set dst (AddReductionVD src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addD_ordered $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ __ vsetvli_helper(T_DOUBLE, Matcher::vector_length(this, $src2)); @@ -2157,7 +2550,6 @@ instruct reduce_addD_unordered(fRegD dst, fRegD src1, vReg src2, vReg tmp) %{ predicate(!n->as_Reduction()->requires_strict_order()); match(Set dst (AddReductionVD src1 src2)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addD_unordered $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ __ vsetvli_helper(T_DOUBLE, Matcher::vector_length(this, $src2)); @@ -2171,14 +2563,13 @@ instruct reduce_addD_unordered(fRegD dst, fRegD src1, vReg src2, vReg tmp) %{ // vector add reduction - predicated -instruct reduce_addI_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ +instruct reduce_add_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (AddReductionVI (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "reduce_addI_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} + format %{ "reduce_add_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2193,7 +2584,6 @@ instruct reduce_addL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0 predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (AddReductionVL (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addL_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -2208,7 +2598,6 @@ instruct reduce_addL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v0 instruct reduce_addF_masked(fRegF dst, fRegF src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ match(Set dst (AddReductionVF (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addF_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ __ vsetvli_helper(T_FLOAT, Matcher::vector_length(this, $src2)); @@ -2223,7 +2612,6 @@ instruct reduce_addF_masked(fRegF dst, fRegF src1, vReg src2, vRegMask_V0 v0, vR instruct reduce_addD_masked(fRegD dst, fRegD src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ match(Set dst (AddReductionVD (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "reduce_addD_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ __ vsetvli_helper(T_DOUBLE, Matcher::vector_length(this, $src2)); @@ -2237,14 +2625,13 @@ instruct reduce_addD_masked(fRegD dst, fRegD src1, vReg src2, vRegMask_V0 v0, vR // vector integer max reduction -instruct vreduce_maxI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ +instruct vreduce_max(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (MaxReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP tmp); - format %{ "vreduce_maxI $dst, $src1, $src2\t# KILL $tmp" %} + format %{ "vreduce_max $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2257,7 +2644,6 @@ instruct vreduce_maxI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ instruct vreduce_maxL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (MaxReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP tmp); format %{ "vreduce_maxL $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ @@ -2271,14 +2657,13 @@ instruct vreduce_maxL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ // vector integer max reduction - predicated -instruct vreduce_maxI_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ +instruct vreduce_max_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (MaxReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "vreduce_maxI_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} + format %{ "vreduce_max_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2293,7 +2678,6 @@ instruct vreduce_maxL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (MaxReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "vreduce_maxL_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -2307,14 +2691,13 @@ instruct vreduce_maxL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v // vector integer min reduction -instruct vreduce_minI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ +instruct vreduce_min(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (MinReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP tmp); - format %{ "vreduce_minI $dst, $src1, $src2\t# KILL $tmp" %} + format %{ "vreduce_min $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2327,7 +2710,6 @@ instruct vreduce_minI(iRegINoSp dst, iRegIorL2I src1, vReg src2, vReg tmp) %{ instruct vreduce_minL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (MinReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP tmp); format %{ "vreduce_minL $dst, $src1, $src2\t# KILL $tmp" %} ins_encode %{ @@ -2341,14 +2723,13 @@ instruct vreduce_minL(iRegLNoSp dst, iRegL src1, vReg src2, vReg tmp) %{ // vector integer min reduction - predicated -instruct vreduce_minI_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ +instruct vreduce_min_masked(iRegINoSp dst, iRegIorL2I src1, vReg src2, vRegMask_V0 v0, vReg tmp) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_BYTE || Matcher::vector_element_basic_type(n->in(2)) == T_SHORT || Matcher::vector_element_basic_type(n->in(2)) == T_INT); match(Set dst (MinReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); - format %{ "vreduce_minI_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} + format %{ "vreduce_min_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); __ reduce_integral_v($dst$$Register, $src1$$Register, @@ -2363,7 +2744,6 @@ instruct vreduce_minL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v predicate(Matcher::vector_element_basic_type(n->in(2)) == T_LONG); match(Set dst (MinReductionV (Binary src1 src2) v0)); effect(TEMP tmp); - ins_cost(VEC_COST); format %{ "vreduce_minL_masked $dst, $src1, $src2, $v0\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this, $src2); @@ -2380,7 +2760,6 @@ instruct vreduce_minL_masked(iRegLNoSp dst, iRegL src1, vReg src2, vRegMask_V0 v instruct vreduce_maxF(fRegF dst, fRegF src1, vReg src2, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_FLOAT); match(Set dst (MaxReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_maxF $dst, $src1, $src2, $tmp1, $tmp2" %} ins_encode %{ @@ -2395,7 +2774,6 @@ instruct vreduce_maxF(fRegF dst, fRegF src1, vReg src2, vReg tmp1, vReg tmp2) %{ instruct vreduce_maxD(fRegD dst, fRegD src1, vReg src2, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_DOUBLE); match(Set dst (MaxReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_maxD $dst, $src1, $src2, $tmp1, $tmp2" %} ins_encode %{ @@ -2412,7 +2790,6 @@ instruct vreduce_maxD(fRegD dst, fRegD src1, vReg src2, vReg tmp1, vReg tmp2) %{ instruct vreduce_maxF_masked(fRegF dst, fRegF src1, vReg src2, vRegMask_V0 v0, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_FLOAT); match(Set dst (MaxReductionV (Binary src1 src2) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_maxF_masked $dst, $src1, $src2, $v0\t# KILL $tmp1, $tmp2" %} ins_encode %{ @@ -2428,7 +2805,6 @@ instruct vreduce_maxF_masked(fRegF dst, fRegF src1, vReg src2, vRegMask_V0 v0, v instruct vreduce_maxD_masked(fRegD dst, fRegD src1, vReg src2, vRegMask_V0 v0, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_DOUBLE); match(Set dst (MaxReductionV (Binary src1 src2) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_maxD_masked $dst, $src1, $src2, $v0\t# KILL $tmp1, $tmp2" %} ins_encode %{ @@ -2446,7 +2822,6 @@ instruct vreduce_maxD_masked(fRegD dst, fRegD src1, vReg src2, vRegMask_V0 v0, v instruct vreduce_minF(fRegF dst, fRegF src1, vReg src2, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_FLOAT); match(Set dst (MinReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_minF $dst, $src1, $src2, $tmp1, $tmp2" %} ins_encode %{ @@ -2461,7 +2836,6 @@ instruct vreduce_minF(fRegF dst, fRegF src1, vReg src2, vReg tmp1, vReg tmp2) %{ instruct vreduce_minD(fRegD dst, fRegD src1, vReg src2, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_DOUBLE); match(Set dst (MinReductionV src1 src2)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_minD $dst, $src1, $src2, $tmp1, $tmp2" %} ins_encode %{ @@ -2478,7 +2852,6 @@ instruct vreduce_minD(fRegD dst, fRegD src1, vReg src2, vReg tmp1, vReg tmp2) %{ instruct vreduce_minF_masked(fRegF dst, fRegF src1, vReg src2, vRegMask_V0 v0, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_FLOAT); match(Set dst (MinReductionV (Binary src1 src2) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_minF_masked $dst, $src1, $src2, $v0\t# KILL $tmp1, $tmp2" %} ins_encode %{ @@ -2494,7 +2867,6 @@ instruct vreduce_minF_masked(fRegF dst, fRegF src1, vReg src2, vRegMask_V0 v0, v instruct vreduce_minD_masked(fRegD dst, fRegD src1, vReg src2, vRegMask_V0 v0, vReg tmp1, vReg tmp2) %{ predicate(Matcher::vector_element_basic_type(n->in(2)) == T_DOUBLE); match(Set dst (MinReductionV (Binary src1 src2) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP tmp1, TEMP tmp2); format %{ "vreduce_minD_masked $dst, $src1, $src2, $v0\t# KILL $tmp1, $tmp2" %} ins_encode %{ @@ -2573,7 +2945,6 @@ instruct reduce_mulL_masked(iRegLNoSp dst, iRegL isrc, vReg vsrc, instruct replicate(vReg dst, iRegIorL2I src) %{ predicate(Matcher::is_non_long_integral_vector(n)); match(Set dst (Replicate src)); - ins_cost(VEC_COST); format %{ "replicate $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -2586,7 +2957,6 @@ instruct replicate(vReg dst, iRegIorL2I src) %{ instruct replicateL(vReg dst, iRegL src) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst (Replicate src)); - ins_cost(VEC_COST); format %{ "replicateL $dst, $src" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -2598,7 +2968,6 @@ instruct replicateL(vReg dst, iRegL src) %{ instruct replicate_imm5(vReg dst, immI5 con) %{ predicate(Matcher::is_non_long_integral_vector(n)); match(Set dst (Replicate con)); - ins_cost(VEC_COST); format %{ "replicate_imm5 $dst, $con" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -2611,7 +2980,6 @@ instruct replicate_imm5(vReg dst, immI5 con) %{ instruct replicateL_imm5(vReg dst, immL5 con) %{ predicate(Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst (Replicate con)); - ins_cost(VEC_COST); format %{ "replicateL_imm5 $dst, $con" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -2620,10 +2988,21 @@ instruct replicateL_imm5(vReg dst, immL5 con) %{ ins_pipe(pipe_slow); %} +instruct replicateHF(vReg dst, fRegF src) %{ + predicate(Matcher::vector_element_basic_type(n) == T_SHORT); + match(Set dst (Replicate src)); + format %{ "replicateHF $dst, $src" %} + ins_encode %{ + assert(UseZvfh, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfmv_v_f(as_VectorRegister($dst$$reg), $src$$FloatRegister); + %} + ins_pipe(pipe_slow); +%} + instruct replicateF(vReg dst, fRegF src) %{ predicate(Matcher::vector_element_basic_type(n) == T_FLOAT); match(Set dst (Replicate src)); - ins_cost(VEC_COST); format %{ "replicateF $dst, $src" %} ins_encode %{ __ vsetvli_helper(T_FLOAT, Matcher::vector_length(this)); @@ -2635,7 +3014,6 @@ instruct replicateF(vReg dst, fRegF src) %{ instruct replicateD(vReg dst, fRegD src) %{ predicate(Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst (Replicate src)); - ins_cost(VEC_COST); format %{ "replicateD $dst, $src" %} ins_encode %{ __ vsetvli_helper(T_DOUBLE, Matcher::vector_length(this)); @@ -2674,7 +3052,6 @@ instruct replicateD(vReg dst, fRegD src) %{ instruct vasrB(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ match(Set dst (RShiftVB src shift)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP v0); format %{ "vasrB $dst, $src, $shift" %} ins_encode %{ @@ -2693,7 +3070,6 @@ instruct vasrB(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ instruct vasrS(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ match(Set dst (RShiftVS src shift)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP v0); format %{ "vasrS $dst, $src, $shift" %} ins_encode %{ @@ -2712,7 +3088,6 @@ instruct vasrS(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ instruct vasrI(vReg dst, vReg src, vReg shift) %{ match(Set dst (RShiftVI src shift)); - ins_cost(VEC_COST); format %{ "vasrI $dst, $src, $shift" %} ins_encode %{ __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -2724,7 +3099,6 @@ instruct vasrI(vReg dst, vReg src, vReg shift) %{ instruct vasrL(vReg dst, vReg src, vReg shift) %{ match(Set dst (RShiftVL src shift)); - ins_cost(VEC_COST); format %{ "vasrL $dst, $src, $shift" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -2736,7 +3110,6 @@ instruct vasrL(vReg dst, vReg src, vReg shift) %{ instruct vasrB_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVB (Binary dst_src shift) vmask)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src, TEMP v0); format %{ "vasrB_masked $dst_src, $dst_src, $shift, $vmask\t# KILL $v0" %} ins_encode %{ @@ -2754,7 +3127,6 @@ instruct vasrB_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) instruct vasrS_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVS (Binary dst_src shift) vmask)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src, TEMP v0); format %{ "vasrS_masked $dst_src, $dst_src, $shift, $vmask\t# KILL $v0" %} ins_encode %{ @@ -2772,7 +3144,6 @@ instruct vasrS_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) instruct vasrI_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVI (Binary dst_src shift) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src); format %{ "vasrI_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ @@ -2785,7 +3156,6 @@ instruct vasrI_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ instruct vasrL_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVL (Binary dst_src shift) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src); format %{ "vasrL_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ @@ -2798,7 +3168,6 @@ instruct vasrL_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ instruct vlslB(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ match(Set dst (LShiftVB src shift)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP v0); format %{ "vlslB $dst, $src, $shift" %} ins_encode %{ @@ -2817,7 +3186,6 @@ instruct vlslB(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ instruct vlslS(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ match(Set dst (LShiftVS src shift)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP v0); format %{ "vlslS $dst, $src, $shift" %} ins_encode %{ @@ -2836,7 +3204,6 @@ instruct vlslS(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ instruct vlslI(vReg dst, vReg src, vReg shift) %{ match(Set dst (LShiftVI src shift)); - ins_cost(VEC_COST); format %{ "vlslI $dst, $src, $shift" %} ins_encode %{ __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -2848,7 +3215,6 @@ instruct vlslI(vReg dst, vReg src, vReg shift) %{ instruct vlslL(vReg dst, vReg src, vReg shift) %{ match(Set dst (LShiftVL src shift)); - ins_cost(VEC_COST); format %{ "vlslL $dst, $src, $shift" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -2860,7 +3226,6 @@ instruct vlslL(vReg dst, vReg src, vReg shift) %{ instruct vlslB_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVB (Binary dst_src shift) vmask)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src, TEMP v0); format %{ "vlslB_masked $dst_src, $dst_src, $shift, $vmask\t# KILL $v0" %} ins_encode %{ @@ -2881,7 +3246,6 @@ instruct vlslB_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) instruct vlslS_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVS (Binary dst_src shift) vmask)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src, TEMP v0); format %{ "vlslS_masked $dst_src, $dst_src, $shift, $vmask\t# KILL $v0" %} ins_encode %{ @@ -2902,7 +3266,6 @@ instruct vlslS_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) instruct vlslI_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVI (Binary dst_src shift) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src); format %{ "vlslI_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ @@ -2915,7 +3278,6 @@ instruct vlslI_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ instruct vlslL_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVL (Binary dst_src shift) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src); format %{ "vlslL_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ @@ -2928,7 +3290,6 @@ instruct vlslL_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ instruct vlsrB(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ match(Set dst (URShiftVB src shift)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP v0); format %{ "vlsrB $dst, $src, $shift" %} ins_encode %{ @@ -2947,7 +3308,6 @@ instruct vlsrB(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ instruct vlsrS(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ match(Set dst (URShiftVS src shift)); - ins_cost(VEC_COST); effect(TEMP_DEF dst, TEMP v0); format %{ "vlsrS $dst, $src, $shift" %} ins_encode %{ @@ -2966,7 +3326,6 @@ instruct vlsrS(vReg dst, vReg src, vReg shift, vRegMask_V0 v0) %{ instruct vlsrI(vReg dst, vReg src, vReg shift) %{ match(Set dst (URShiftVI src shift)); - ins_cost(VEC_COST); format %{ "vlsrI $dst, $src, $shift" %} ins_encode %{ __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -2978,7 +3337,6 @@ instruct vlsrI(vReg dst, vReg src, vReg shift) %{ instruct vlsrL(vReg dst, vReg src, vReg shift) %{ match(Set dst (URShiftVL src shift)); - ins_cost(VEC_COST); format %{ "vlsrL $dst, $src, $shift" %} ins_encode %{ __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -2990,7 +3348,6 @@ instruct vlsrL(vReg dst, vReg src, vReg shift) %{ instruct vlsrB_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVB (Binary dst_src shift) vmask)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src, TEMP v0); format %{ "vlsrB_masked $dst_src, $dst_src, $shift, $vmask\t# KILL $v0" %} ins_encode %{ @@ -3011,7 +3368,6 @@ instruct vlsrB_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) instruct vlsrS_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVS (Binary dst_src shift) vmask)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src, TEMP v0); format %{ "vlsrS_masked $dst_src, $dst_src, $shift, $vmask\t# KILL $v0" %} ins_encode %{ @@ -3032,7 +3388,6 @@ instruct vlsrS_masked(vReg dst_src, vReg shift, vRegMask vmask, vRegMask_V0 v0) instruct vlsrI_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVI (Binary dst_src shift) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src); format %{ "vlsrI_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ @@ -3045,7 +3400,6 @@ instruct vlsrI_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ instruct vlsrL_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVL (Binary dst_src shift) v0)); - ins_cost(VEC_COST); effect(TEMP_DEF dst_src); format %{ "vlsrL_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ @@ -3056,10 +3410,9 @@ instruct vlsrL_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vasrB_imm(vReg dst, vReg src, immI shift) %{ +instruct vasrB_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (RShiftVB src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vasrB_imm $dst, $src, $shift" %} + format %{ "vasrB_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); @@ -3074,10 +3427,9 @@ instruct vasrB_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vasrS_imm(vReg dst, vReg src, immI shift) %{ +instruct vasrS_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (RShiftVS src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vasrS_imm $dst, $src, $shift" %} + format %{ "vasrS_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); @@ -3092,10 +3444,9 @@ instruct vasrS_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vasrI_imm(vReg dst, vReg src, immI shift) %{ +instruct vasrI_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (RShiftVI src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vasrI_imm $dst, $src, $shift" %} + format %{ "vasrI_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -3109,11 +3460,10 @@ instruct vasrI_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vasrL_imm(vReg dst, vReg src, immI shift) %{ +instruct vasrL_vi(vReg dst, vReg src, immI shift) %{ predicate((n->in(2)->in(1)->get_int() & 0x3f) < 32); match(Set dst (RShiftVL src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vasrL_imm $dst, $src, $shift" %} + format %{ "vasrL_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -3127,10 +3477,9 @@ instruct vasrL_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vasrB_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vasrB_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVB (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vasrB_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vasrB_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3144,10 +3493,9 @@ instruct vasrB_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vasrS_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vasrS_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVS (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vasrS_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vasrS_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3161,10 +3509,9 @@ instruct vasrS_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vasrI_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vasrI_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (RShiftVI (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vasrI_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vasrI_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3177,11 +3524,10 @@ instruct vasrI_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vasrL_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vasrL_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ predicate((n->in(1)->in(2)->in(1)->get_int() & 0x3f) < 32); match(Set dst_src (RShiftVL (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vasrL_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vasrL_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3194,10 +3540,9 @@ instruct vasrL_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlsrB_imm(vReg dst, vReg src, immI shift) %{ +instruct vlsrB_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (URShiftVB src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlsrB_imm $dst, $src, $shift" %} + format %{ "vlsrB_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); @@ -3216,10 +3561,9 @@ instruct vlsrB_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlsrS_imm(vReg dst, vReg src, immI shift) %{ +instruct vlsrS_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (URShiftVS src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlsrS_imm $dst, $src, $shift" %} + format %{ "vlsrS_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); @@ -3238,10 +3582,9 @@ instruct vlsrS_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlsrI_imm(vReg dst, vReg src, immI shift) %{ +instruct vlsrI_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (URShiftVI src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlsrI_imm $dst, $src, $shift" %} + format %{ "vlsrI_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -3255,11 +3598,10 @@ instruct vlsrI_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlsrL_imm(vReg dst, vReg src, immI shift) %{ +instruct vlsrL_vi(vReg dst, vReg src, immI shift) %{ predicate((n->in(2)->in(1)->get_int() & 0x3f) < 32); match(Set dst (URShiftVL src (RShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlsrL_imm $dst, $src, $shift" %} + format %{ "vlsrL_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -3273,10 +3615,9 @@ instruct vlsrL_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlsrB_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlsrB_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVB (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlsrB_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlsrB_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3294,10 +3635,9 @@ instruct vlsrB_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlsrS_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlsrS_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVS (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlsrS_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlsrS_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3315,10 +3655,9 @@ instruct vlsrS_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlsrI_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlsrI_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (URShiftVI (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlsrI_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlsrI_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3331,11 +3670,10 @@ instruct vlsrI_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlsrL_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlsrL_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ predicate((n->in(1)->in(2)->in(1)->get_int() & 0x3f) < 32); match(Set dst_src (URShiftVL (Binary dst_src (RShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlsrL_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlsrL_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; if (con == 0) { @@ -3348,10 +3686,9 @@ instruct vlsrL_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlslB_imm(vReg dst, vReg src, immI shift) %{ +instruct vlslB_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (LShiftVB src (LShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlslB_imm $dst, $src, $shift" %} + format %{ "vlslB_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); @@ -3365,10 +3702,9 @@ instruct vlslB_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlslS_imm(vReg dst, vReg src, immI shift) %{ +instruct vlslS_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (LShiftVS src (LShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlslS_imm $dst, $src, $shift" %} + format %{ "vlslS_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); @@ -3382,10 +3718,9 @@ instruct vlslS_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlslI_imm(vReg dst, vReg src, immI shift) %{ +instruct vlslI_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (LShiftVI src (LShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlslI_imm $dst, $src, $shift" %} + format %{ "vlslI_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -3394,11 +3729,10 @@ instruct vlslI_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlslL_imm(vReg dst, vReg src, immI shift) %{ +instruct vlslL_vi(vReg dst, vReg src, immI shift) %{ predicate((n->in(2)->in(1)->get_int() & 0x3f) < 32); match(Set dst (LShiftVL src (LShiftCntV shift))); - ins_cost(VEC_COST); - format %{ "vlslL_imm $dst, $src, $shift" %} + format %{ "vlslL_vi $dst, $src, $shift" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -3407,10 +3741,9 @@ instruct vlslL_imm(vReg dst, vReg src, immI shift) %{ ins_pipe(pipe_slow); %} -instruct vlslB_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlslB_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVB (Binary dst_src (LShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlslB_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlslB_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_BYTE, Matcher::vector_length(this)); @@ -3425,10 +3758,9 @@ instruct vlslB_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlslS_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlslS_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVS (Binary dst_src (LShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlslS_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlslS_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); @@ -3443,10 +3775,9 @@ instruct vlslS_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlslI_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlslI_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (LShiftVI (Binary dst_src (LShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlslI_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlslI_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_INT, Matcher::vector_length(this)); @@ -3456,11 +3787,10 @@ instruct vlslI_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vlslL_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vlslL_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ predicate((n->in(1)->in(2)->in(1)->get_int() & 0x3f) < 32); match(Set dst_src (LShiftVL (Binary dst_src (LShiftCntV shift)) v0)); - ins_cost(VEC_COST); - format %{ "vlslL_imm_masked $dst_src, $dst_src, $shift, $v0" %} + format %{ "vlslL_vi_masked $dst_src, $dst_src, $shift, $v0" %} ins_encode %{ uint32_t con = (unsigned)$shift$$constant & 0x1f; __ vsetvli_helper(T_LONG, Matcher::vector_length(this)); @@ -3499,9 +3829,10 @@ instruct vrotate_right(vReg dst, vReg src, vReg shift) %{ ins_pipe(pipe_slow); %} -instruct vrotate_right_reg(vReg dst, vReg src, iRegIorL2I shift) %{ +// Only the low log2(SEW) bits of shift value are used, all other bits are ignored. +instruct vrotate_right_vx(vReg dst, vReg src, iRegIorL2I shift) %{ match(Set dst (RotateRightV src (Replicate shift))); - format %{ "vrotate_right_reg $dst, $src, $shift\t" %} + format %{ "vrotate_right_vx $dst, $src, $shift\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -3511,9 +3842,9 @@ instruct vrotate_right_reg(vReg dst, vReg src, iRegIorL2I shift) %{ ins_pipe(pipe_slow); %} -instruct vrotate_right_imm(vReg dst, vReg src, immI shift) %{ +instruct vrotate_right_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (RotateRightV src shift)); - format %{ "vrotate_right_imm $dst, $src, $shift\t" %} + format %{ "vrotate_right_vi $dst, $src, $shift\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); uint32_t bits = type2aelembytes(bt) * 8; @@ -3531,7 +3862,7 @@ instruct vrotate_right_imm(vReg dst, vReg src, immI shift) %{ instruct vrotate_right_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (RotateRightV (Binary dst_src shift) v0)); - format %{ "vrotate_right_masked $dst_src, $dst_src, $shift, v0.t\t" %} + format %{ "vrotate_right_masked $dst_src, $dst_src, $shift, $v0\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -3541,9 +3872,10 @@ instruct vrotate_right_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vrotate_right_reg_masked(vReg dst_src, iRegIorL2I shift, vRegMask_V0 v0) %{ +// Only the low log2(SEW) bits of shift value are used, all other bits are ignored. +instruct vrotate_right_vx_masked(vReg dst_src, iRegIorL2I shift, vRegMask_V0 v0) %{ match(Set dst_src (RotateRightV (Binary dst_src (Replicate shift)) v0)); - format %{ "vrotate_right_reg_masked $dst_src, $dst_src, $shift, v0.t\t" %} + format %{ "vrotate_right_vx_masked $dst_src, $dst_src, $shift, $v0\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -3553,9 +3885,9 @@ instruct vrotate_right_reg_masked(vReg dst_src, iRegIorL2I shift, vRegMask_V0 v0 ins_pipe(pipe_slow); %} -instruct vrotate_right_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vrotate_right_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (RotateRightV (Binary dst_src shift) v0)); - format %{ "vrotate_right_imm_masked $dst_src, $dst_src, $shift, v0.t\t" %} + format %{ "vrotate_right_vi_masked $dst_src, $dst_src, $shift, $v0\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); uint32_t bits = type2aelembytes(bt) * 8; @@ -3584,9 +3916,10 @@ instruct vrotate_left(vReg dst, vReg src, vReg shift) %{ ins_pipe(pipe_slow); %} -instruct vrotate_left_reg(vReg dst, vReg src, iRegIorL2I shift) %{ +// Only the low log2(SEW) bits of shift value are used, all other bits are ignored. +instruct vrotate_left_vx(vReg dst, vReg src, iRegIorL2I shift) %{ match(Set dst (RotateLeftV src (Replicate shift))); - format %{ "vrotate_left_reg $dst, $src, $shift\t" %} + format %{ "vrotate_left_vx $dst, $src, $shift\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -3596,9 +3929,9 @@ instruct vrotate_left_reg(vReg dst, vReg src, iRegIorL2I shift) %{ ins_pipe(pipe_slow); %} -instruct vrotate_left_imm(vReg dst, vReg src, immI shift) %{ +instruct vrotate_left_vi(vReg dst, vReg src, immI shift) %{ match(Set dst (RotateLeftV src shift)); - format %{ "vrotate_left_imm $dst, $src, $shift\t" %} + format %{ "vrotate_left_vi $dst, $src, $shift\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); uint32_t bits = type2aelembytes(bt) * 8; @@ -3617,7 +3950,7 @@ instruct vrotate_left_imm(vReg dst, vReg src, immI shift) %{ instruct vrotate_left_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ match(Set dst_src (RotateLeftV (Binary dst_src shift) v0)); - format %{ "vrotate_left_masked $dst_src, $dst_src, $shift, v0.t\t" %} + format %{ "vrotate_left_masked $dst_src, $dst_src, $shift, $v0\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -3627,9 +3960,10 @@ instruct vrotate_left_masked(vReg dst_src, vReg shift, vRegMask_V0 v0) %{ ins_pipe(pipe_slow); %} -instruct vrotate_left_reg_masked(vReg dst_src, iRegIorL2I shift, vRegMask_V0 v0) %{ +// Only the low log2(SEW) bits of shift value are used, all other bits are ignored. +instruct vrotate_left_vx_masked(vReg dst_src, iRegIorL2I shift, vRegMask_V0 v0) %{ match(Set dst_src (RotateLeftV (Binary dst_src (Replicate shift)) v0)); - format %{ "vrotate_left_reg_masked $dst_src, $dst_src, $shift, v0.t\t" %} + format %{ "vrotate_left_vx_masked $dst_src, $dst_src, $shift, $v0\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -3639,9 +3973,9 @@ instruct vrotate_left_reg_masked(vReg dst_src, iRegIorL2I shift, vRegMask_V0 v0) ins_pipe(pipe_slow); %} -instruct vrotate_left_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ +instruct vrotate_left_vi_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ match(Set dst_src (RotateLeftV (Binary dst_src shift) v0)); - format %{ "vrotate_left_imm_masked $dst_src, $dst_src, $shift, v0.t\t" %} + format %{ "vrotate_left_vi_masked $dst_src, $dst_src, $shift, $v0\t" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); uint32_t bits = type2aelembytes(bt) * 8; @@ -3659,10 +3993,21 @@ instruct vrotate_left_imm_masked(vReg dst_src, immI shift, vRegMask_V0 v0) %{ // vector sqrt +instruct vsqrt_hfp(vReg dst, vReg src) %{ + match(Set dst (SqrtVHF src)); + format %{ "vsqrt_hfp $dst, $src" %} + ins_encode %{ + assert(UseZvfh, "must"); + assert(Matcher::vector_element_basic_type(this) == T_SHORT, "must"); + __ vsetvli_helper(T_SHORT, Matcher::vector_length(this)); + __ vfsqrt_v(as_VectorRegister($dst$$reg), as_VectorRegister($src$$reg)); + %} + ins_pipe(pipe_slow); +%} + instruct vsqrt_fp(vReg dst, vReg src) %{ match(Set dst (SqrtVF src)); match(Set dst (SqrtVD src)); - ins_cost(VEC_COST); format %{ "vsqrt_fp $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -3677,7 +4022,6 @@ instruct vsqrt_fp(vReg dst, vReg src) %{ instruct vsqrt_fp_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (SqrtVF dst_src v0)); match(Set dst_src (SqrtVD dst_src v0)); - ins_cost(VEC_COST); format %{ "vsqrt_fp_masked $dst_src, $dst_src, $v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -3976,7 +4320,6 @@ instruct vclearArray_reg_reg(iRegL_R29 cnt, iRegP_R28 base, Universe dummy, // Vector Load Const instruct vloadcon(vReg dst, immI0 src) %{ match(Set dst (VectorLoadConst src)); - ins_cost(VEC_COST); format %{ "vloadcon $dst\t# generate iota indices" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -4237,8 +4580,8 @@ instruct vcvtStoB(vReg dst, vReg src) %{ %} instruct vcvtStoX(vReg dst, vReg src) %{ - predicate((Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_LONG)); + predicate(Matcher::vector_element_basic_type(n) == T_INT || + Matcher::vector_element_basic_type(n) == T_LONG); match(Set dst (VectorCastS2X src)); effect(TEMP_DEF dst); format %{ "vcvtStoX $dst, $src" %} @@ -4251,8 +4594,8 @@ instruct vcvtStoX(vReg dst, vReg src) %{ %} instruct vcvtStoX_fp(vReg dst, vReg src) %{ - predicate((Matcher::vector_element_basic_type(n) == T_FLOAT || - Matcher::vector_element_basic_type(n) == T_DOUBLE)); + predicate(Matcher::vector_element_basic_type(n) == T_FLOAT || + Matcher::vector_element_basic_type(n) == T_DOUBLE); match(Set dst (VectorCastS2X src)); effect(TEMP_DEF dst); format %{ "vcvtStoX_fp $dst, $src" %} @@ -4349,9 +4692,9 @@ instruct vcvtItoD(vReg dst, vReg src) %{ // VectorCastL2X instruct vcvtLtoI(vReg dst, vReg src) %{ - predicate(Matcher::vector_element_basic_type(n) == T_INT || - Matcher::vector_element_basic_type(n) == T_BYTE || - Matcher::vector_element_basic_type(n) == T_SHORT); + predicate(Matcher::vector_element_basic_type(n) == T_BYTE || + Matcher::vector_element_basic_type(n) == T_SHORT || + Matcher::vector_element_basic_type(n) == T_INT); match(Set dst (VectorCastL2X src)); format %{ "vcvtLtoI $dst, $src" %} ins_encode %{ @@ -4845,7 +5188,6 @@ instruct vconvF2HF(vReg dst, vReg src, vReg vtmp, vRegMask_V0 v0, iRegINoSp tmp) instruct vpopcount_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (PopCountVI dst_src v0)); match(Set dst_src (PopCountVL dst_src v0)); - ins_cost(VEC_COST); format %{ "vcpop_v $dst_src, $dst_src, $v0\t# vcpop_v with mask" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -4859,7 +5201,6 @@ instruct vpopcount_masked(vReg dst_src, vRegMask_V0 v0) %{ instruct vpopcount(vReg dst, vReg src) %{ match(Set dst (PopCountVI src)); match(Set dst (PopCountVL src)); - ins_cost(VEC_COST); format %{ "vcpop_v $dst, $src\t# vcpop_v without mask" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -4874,7 +5215,6 @@ instruct vpopcount(vReg dst, vReg src) %{ instruct vcountLeadingZeros_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (CountLeadingZerosV dst_src v0)); - ins_cost(VEC_COST); format %{ "vcount_leading_zeros_masked $dst_src, $dst_src, v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -4887,7 +5227,6 @@ instruct vcountLeadingZeros_masked(vReg dst_src, vRegMask_V0 v0) %{ instruct vcountLeadingZeros(vReg dst, vReg src) %{ match(Set dst (CountLeadingZerosV src)); - ins_cost(VEC_COST); format %{ "vcount_leading_zeros $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -4902,7 +5241,6 @@ instruct vcountLeadingZeros(vReg dst, vReg src) %{ instruct vcountTrailingZeros_masked(vReg dst_src, vRegMask_V0 v0) %{ match(Set dst_src (CountTrailingZerosV dst_src v0)); - ins_cost(VEC_COST); format %{ "vcount_trailing_zeros_masked $dst_src, $dst_src, v0" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -4915,7 +5253,6 @@ instruct vcountTrailingZeros_masked(vReg dst_src, vRegMask_V0 v0) %{ instruct vcountTrailingZeros(vReg dst, vReg src) %{ match(Set dst (CountTrailingZerosV src)); - ins_cost(VEC_COST); format %{ "vcount_trailing_zeros $dst, $src" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); @@ -5087,14 +5424,14 @@ instruct populateindex(vReg dst, iRegIorL2I src1, iRegIorL2I src2, vReg tmp) %{ // BYTE, SHORT, INT -instruct insertI_index_lt32(vReg dst, vReg src, iRegIorL2I val, immI idx, vRegMask_V0 v0) %{ +instruct insert_index_lt32(vReg dst, vReg src, iRegIorL2I val, immI idx, vRegMask_V0 v0) %{ predicate(n->in(2)->get_int() < 32 && (Matcher::vector_element_basic_type(n) == T_BYTE || Matcher::vector_element_basic_type(n) == T_SHORT || Matcher::vector_element_basic_type(n) == T_INT)); match(Set dst (VectorInsert (Binary src val) idx)); effect(TEMP v0); - format %{ "insertI_index_lt32 $dst, $src, $val, $idx" %} + format %{ "insert_index_lt32 $dst, $src, $val, $idx" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); @@ -5106,14 +5443,14 @@ instruct insertI_index_lt32(vReg dst, vReg src, iRegIorL2I val, immI idx, vRegMa ins_pipe(pipe_slow); %} -instruct insertI_index(vReg dst, vReg src, iRegIorL2I val, iRegIorL2I idx, vReg tmp, vRegMask_V0 v0) %{ +instruct insert_index(vReg dst, vReg src, iRegIorL2I val, iRegIorL2I idx, vReg tmp, vRegMask_V0 v0) %{ predicate(n->in(2)->get_int() >= 32 && (Matcher::vector_element_basic_type(n) == T_BYTE || Matcher::vector_element_basic_type(n) == T_SHORT || Matcher::vector_element_basic_type(n) == T_INT)); match(Set dst (VectorInsert (Binary src val) idx)); effect(TEMP tmp, TEMP v0); - format %{ "insertI_index $dst, $src, $val, $idx\t# KILL $tmp" %} + format %{ "insert_index $dst, $src, $val, $idx\t# KILL $tmp" %} ins_encode %{ BasicType bt = Matcher::vector_element_basic_type(this); __ vsetvli_helper(bt, Matcher::vector_length(this)); diff --git a/src/hotspot/cpu/riscv/runtime_riscv.cpp b/src/hotspot/cpu/riscv/runtime_riscv.cpp index 44a8e35e285..7c8ca853bc4 100644 --- a/src/hotspot/cpu/riscv/runtime_riscv.cpp +++ b/src/hotspot/cpu/riscv/runtime_riscv.cpp @@ -63,6 +63,9 @@ UncommonTrapBlob* OptoRuntime::generate_uncommon_trap_blob() { // Setup code generation tools const char* name = OptoRuntime::stub_name(OptoStubId::uncommon_trap_id); CodeBuffer buffer(name, 2048, 1024); + if (buffer.blob() == nullptr) { + return nullptr; + } MacroAssembler* masm = new MacroAssembler(&buffer); assert_cond(masm != nullptr); @@ -282,6 +285,9 @@ ExceptionBlob* OptoRuntime::generate_exception_blob() { // Setup code generation tools const char* name = OptoRuntime::stub_name(OptoStubId::exception_id); CodeBuffer buffer(name, 2048, 1024); + if (buffer.blob() == nullptr) { + return nullptr; + } MacroAssembler* masm = new MacroAssembler(&buffer); assert_cond(masm != nullptr); diff --git a/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp b/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp index 10790841490..391be81c1ae 100644 --- a/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp +++ b/src/hotspot/cpu/riscv/sharedRuntime_riscv.cpp @@ -596,12 +596,13 @@ void SharedRuntime::gen_i2c_adapter(MacroAssembler *masm, } // --------------------------------------------------------------- -AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm, - int total_args_passed, - int comp_args_on_stack, - const BasicType *sig_bt, - const VMRegPair *regs, - AdapterFingerPrint* fingerprint) { + +void SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm, + int total_args_passed, + int comp_args_on_stack, + const BasicType *sig_bt, + const VMRegPair *regs, + AdapterHandlerEntry* handler) { address i2c_entry = __ pc(); gen_i2c_adapter(masm, total_args_passed, comp_args_on_stack, sig_bt, regs); @@ -658,7 +659,8 @@ AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm gen_c2i_adapter(masm, total_args_passed, comp_args_on_stack, sig_bt, regs, skip_fixup); - return AdapterHandlerLibrary::new_entry(fingerprint, i2c_entry, c2i_entry, c2i_unverified_entry, c2i_no_clinit_check_entry); + handler->set_entry_points(i2c_entry, c2i_entry, c2i_unverified_entry, c2i_no_clinit_check_entry); + return; } int SharedRuntime::vector_calling_convention(VMRegPair *regs, @@ -1323,7 +1325,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm, // First instruction must be a nop as it may need to be patched on deoptimisation { - Assembler::IncompressibleRegion ir(masm); // keep the nop as 4 bytes for patching. + Assembler::IncompressibleScope scope(masm); // keep the nop as 4 bytes for patching. MacroAssembler::assert_alignment(__ pc()); __ nop(); // 4 bytes } @@ -1466,7 +1468,7 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm, // If we have to make this method not-entrant we'll overwrite its // first instruction with a jump. { - Assembler::IncompressibleRegion ir(masm); // keep the nop as 4 bytes for patching. + Assembler::IncompressibleScope scope(masm); // keep the nop as 4 bytes for patching. MacroAssembler::assert_alignment(__ pc()); __ nop(); // 4 bytes } @@ -1892,6 +1894,24 @@ nmethod* SharedRuntime::generate_native_wrapper(MacroAssembler* masm, __ leave(); + #if INCLUDE_JFR + // We need to do a poll test after unwind in case the sampler + // managed to sample the native frame after returning to Java. + Label L_return; + __ ld(t0, Address(xthread, JavaThread::polling_word_offset())); + address poll_test_pc = __ pc(); + __ relocate(relocInfo::poll_return_type); + __ test_bit(t0, t0, log2i_exact(SafepointMechanism::poll_bit())); + __ beqz(t0, L_return); + assert(SharedRuntime::polling_page_return_handler_blob() != nullptr, + "polling page return stub not created yet"); + address stub = SharedRuntime::polling_page_return_handler_blob()->entry_point(); + __ la(t0, InternalAddress(poll_test_pc)); + __ sd(t0, Address(xthread, JavaThread::saved_exception_pc_offset())); + __ far_jump(RuntimeAddress(stub)); + __ bind(L_return); +#endif // INCLUDE_JFR + // Any exception pending? Label exception_pending; __ ld(t0, Address(xthread, in_bytes(Thread::pending_exception_offset()))); diff --git a/src/hotspot/cpu/riscv/stubGenerator_riscv.cpp b/src/hotspot/cpu/riscv/stubGenerator_riscv.cpp index 4527a32926f..c58f6bc338d 100644 --- a/src/hotspot/cpu/riscv/stubGenerator_riscv.cpp +++ b/src/hotspot/cpu/riscv/stubGenerator_riscv.cpp @@ -1,7 +1,7 @@ /* * Copyright (c) 2003, 2025, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2014, 2025, Red Hat Inc. All rights reserved. - * Copyright (c) 2020, 2023, Huawei Technologies Co., Ltd. All rights reserved. + * Copyright (c) 2020, 2025, Huawei Technologies Co., Ltd. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -886,7 +886,7 @@ class StubGenerator: public StubCodeGenerator { void copy_memory_v(Register s, Register d, Register count, int step) { bool is_backward = step < 0; - int granularity = uabs(step); + int granularity = g_uabs(step); const Register src = x30, dst = x31, vl = x14, cnt = x15, tmp1 = x16, tmp2 = x17; assert_different_registers(s, d, cnt, vl, tmp1, tmp2); @@ -948,7 +948,7 @@ class StubGenerator: public StubCodeGenerator { } bool is_backwards = step < 0; - int granularity = uabs(step); + int granularity = g_uabs(step); const Register src = x30, dst = x31, cnt = x15, tmp3 = x16, tmp4 = x17, tmp5 = x14, tmp6 = x13; const Register gct1 = x28, gct2 = x29, gct3 = t2; @@ -1633,6 +1633,126 @@ class StubGenerator: public StubCodeGenerator { BLOCK_COMMENT("arraycopy_range_checks done"); } + address generate_unsafecopy_common_error_exit() { + address start = __ pc(); + __ mv(x10, 0); + __ leave(); + __ ret(); + return start; + } + + // + // Generate 'unsafe' set memory stub + // Though just as safe as the other stubs, it takes an unscaled + // size_t (# bytes) argument instead of an element count. + // + // Input: + // c_rarg0 - destination array address + // c_rarg1 - byte count (size_t) + // c_rarg2 - byte value + // + address generate_unsafe_setmemory() { + __ align(CodeEntryAlignment); + StubGenStubId stub_id = StubGenStubId::unsafe_setmemory_id; + StubCodeMark mark(this, stub_id); + address start = __ pc(); + + // bump this on entry, not on exit: + // inc_counter_np(SharedRuntime::_unsafe_set_memory_ctr); + + Label L_fill_elements; + + const Register dest = c_rarg0; + const Register count = c_rarg1; + const Register value = c_rarg2; + const Register cnt_words = x28; // temp register + const Register tmp_reg = x29; // temp register + + // Mark remaining code as such which performs Unsafe accesses. + UnsafeMemoryAccessMark umam(this, true, false); + + __ enter(); // required for proper stackwalking of RuntimeStub frame + + // if count < 8, jump to L_fill_elements + __ mv(tmp_reg, 8); // 8 bytes fill by element + __ bltu(count, tmp_reg, L_fill_elements); + + // Propagate byte to 64-bit width + // 8 bit -> 16 bit + __ zext(value, value, 8); + __ slli(tmp_reg, value, 8); + __ orr(value, value, tmp_reg); + // 16 bit -> 32 bit + __ slli(tmp_reg, value, 16); + __ orr(value, value, tmp_reg); + // 32 bit -> 64 bit + __ slli(tmp_reg, value, 32); + __ orr(value, value, tmp_reg); + + // Align source address at 8 bytes address boundary. + Label L_skip_align1, L_skip_align2, L_skip_align4; + // One byte misalignment happens. + __ test_bit(tmp_reg, dest, 0); + __ beqz(tmp_reg, L_skip_align1); + __ sb(value, Address(dest, 0)); + __ addi(dest, dest, 1); + __ subi(count, count, 1); + + __ bind(L_skip_align1); + // Two bytes misalignment happens. + __ test_bit(tmp_reg, dest, 1); + __ beqz(tmp_reg, L_skip_align2); + __ sh(value, Address(dest, 0)); + __ addi(dest, dest, 2); + __ subi(count, count, 2); + + __ bind(L_skip_align2); + // Four bytes misalignment happens. + __ test_bit(tmp_reg, dest, 2); + __ beqz(tmp_reg, L_skip_align4); + __ sw(value, Address(dest, 0)); + __ addi(dest, dest, 4); + __ subi(count, count, 4); + __ bind(L_skip_align4); + + // Fill large chunks + __ srli(cnt_words, count, 3); // number of words + __ slli(tmp_reg, cnt_words, 3); + __ sub(count, count, tmp_reg); + { + __ fill_words(dest, cnt_words, value); + } + + // Handle copies less than 8 bytes + __ bind(L_fill_elements); + Label L_fill_2, L_fill_1, L_exit; + __ test_bit(tmp_reg, count, 2); + __ beqz(tmp_reg, L_fill_2); + __ sb(value, Address(dest, 0)); + __ sb(value, Address(dest, 1)); + __ sb(value, Address(dest, 2)); + __ sb(value, Address(dest, 3)); + __ addi(dest, dest, 4); + + __ bind(L_fill_2); + __ test_bit(tmp_reg, count, 1); + __ beqz(tmp_reg, L_fill_1); + __ sb(value, Address(dest, 0)); + __ sb(value, Address(dest, 1)); + __ addi(dest, dest, 2); + + __ bind(L_fill_1); + __ test_bit(tmp_reg, count, 0); + __ beqz(tmp_reg, L_exit); + __ sb(value, Address(dest, 0)); + + __ bind(L_exit); + __ leave(); + __ ret(); + + return start; + } + // // Generate 'unsafe' array copy stub // Though just as safe as the other stubs, it takes an unscaled @@ -2029,44 +2149,40 @@ class StubGenerator: public StubCodeGenerator { __ enter(); - Label L_fill_elements, L_exit1; + Label L_fill_elements; int shift = -1; switch (t) { case T_BYTE: shift = 0; + // Short arrays (< 8 bytes) fill by element + __ mv(tmp_reg, 8 >> shift); + __ bltu(count, tmp_reg, L_fill_elements); // Zero extend value // 8 bit -> 16 bit __ zext(value, value, 8); - __ mv(tmp_reg, value); - __ slli(tmp_reg, tmp_reg, 8); + __ slli(tmp_reg, value, 8); __ orr(value, value, tmp_reg); // 16 bit -> 32 bit - __ mv(tmp_reg, value); - __ slli(tmp_reg, tmp_reg, 16); + __ slli(tmp_reg, value, 16); __ orr(value, value, tmp_reg); - - __ mv(tmp_reg, 8 >> shift); // Short arrays (< 8 bytes) fill by element - __ bltu(count, tmp_reg, L_fill_elements); break; case T_SHORT: shift = 1; + // Short arrays (< 8 bytes) fill by element + __ mv(tmp_reg, 8 >> shift); + __ bltu(count, tmp_reg, L_fill_elements); + // Zero extend value // 16 bit -> 32 bit __ zext(value, value, 16); - __ mv(tmp_reg, value); - __ slli(tmp_reg, tmp_reg, 16); + __ slli(tmp_reg, value, 16); __ orr(value, value, tmp_reg); - - // Short arrays (< 8 bytes) fill by element - __ mv(tmp_reg, 8 >> shift); - __ bltu(count, tmp_reg, L_fill_elements); break; case T_INT: shift = 2; - // Short arrays (< 8 bytes) fill by element __ mv(tmp_reg, 8 >> shift); __ bltu(count, tmp_reg, L_fill_elements); @@ -2080,8 +2196,8 @@ class StubGenerator: public StubCodeGenerator { switch (t) { case T_BYTE: // One byte misalignment happens only for byte arrays. - __ test_bit(t0, to, 0); - __ beqz(t0, L_skip_align1); + __ test_bit(tmp_reg, to, 0); + __ beqz(tmp_reg, L_skip_align1); __ sb(value, Address(to, 0)); __ addi(to, to, 1); __ subiw(count, count, 1); @@ -2089,8 +2205,8 @@ class StubGenerator: public StubCodeGenerator { // Fallthrough case T_SHORT: // Two bytes misalignment happens only for byte and short (char) arrays. - __ test_bit(t0, to, 1); - __ beqz(t0, L_skip_align2); + __ test_bit(tmp_reg, to, 1); + __ beqz(tmp_reg, L_skip_align2); __ sh(value, Address(to, 0)); __ addi(to, to, 2); __ subiw(count, count, 2 >> shift); @@ -2098,8 +2214,8 @@ class StubGenerator: public StubCodeGenerator { // Fallthrough case T_INT: // Align to 8 bytes, we know we are 4 byte aligned to start. - __ test_bit(t0, to, 2); - __ beqz(t0, L_skip_align4); + __ test_bit(tmp_reg, to, 2); + __ beqz(tmp_reg, L_skip_align4); __ sw(value, Address(to, 0)); __ addi(to, to, 4); __ subiw(count, count, 4 >> shift); @@ -2125,55 +2241,54 @@ class StubGenerator: public StubCodeGenerator { __ fill_words(to, cnt_words, value); } - // Remaining count is less than 8 bytes. Fill it by a single store. - // Note that the total length is no less than 8 bytes. - if (!AvoidUnalignedAccesses && (t == T_BYTE || t == T_SHORT)) { - __ beqz(count, L_exit1); - __ shadd(to, count, to, tmp_reg, shift); // points to the end - __ sd(value, Address(to, -8)); // overwrite some elements - __ bind(L_exit1); - __ leave(); - __ ret(); - } - // Handle copies less than 8 bytes. - Label L_fill_2, L_fill_4, L_exit2; + // Address may not be heapword aligned. + Label L_fill_1, L_fill_2, L_exit; __ bind(L_fill_elements); switch (t) { case T_BYTE: - __ test_bit(t0, count, 0); - __ beqz(t0, L_fill_2); + __ test_bit(tmp_reg, count, 2); + __ beqz(tmp_reg, L_fill_2); __ sb(value, Address(to, 0)); - __ addi(to, to, 1); + __ sb(value, Address(to, 1)); + __ sb(value, Address(to, 2)); + __ sb(value, Address(to, 3)); + __ addi(to, to, 4); + __ bind(L_fill_2); - __ test_bit(t0, count, 1); - __ beqz(t0, L_fill_4); - __ sh(value, Address(to, 0)); + __ test_bit(tmp_reg, count, 1); + __ beqz(tmp_reg, L_fill_1); + __ sb(value, Address(to, 0)); + __ sb(value, Address(to, 1)); __ addi(to, to, 2); - __ bind(L_fill_4); - __ test_bit(t0, count, 2); - __ beqz(t0, L_exit2); - __ sw(value, Address(to, 0)); + + __ bind(L_fill_1); + __ test_bit(tmp_reg, count, 0); + __ beqz(tmp_reg, L_exit); + __ sb(value, Address(to, 0)); break; case T_SHORT: - __ test_bit(t0, count, 0); - __ beqz(t0, L_fill_4); + __ test_bit(tmp_reg, count, 1); + __ beqz(tmp_reg, L_fill_2); + __ sh(value, Address(to, 0)); + __ sh(value, Address(to, 2)); + __ addi(to, to, 4); + + __ bind(L_fill_2); + __ test_bit(tmp_reg, count, 0); + __ beqz(tmp_reg, L_exit); __ sh(value, Address(to, 0)); - __ addi(to, to, 2); - __ bind(L_fill_4); - __ test_bit(t0, count, 1); - __ beqz(t0, L_exit2); - __ sw(value, Address(to, 0)); break; case T_INT: - __ beqz(count, L_exit2); + __ beqz(count, L_exit); __ sw(value, Address(to, 0)); break; default: ShouldNotReachHere(); } - __ bind(L_exit2); + __ bind(L_exit); __ leave(); __ ret(); + return start; } @@ -2189,6 +2304,9 @@ class StubGenerator: public StubCodeGenerator { generate_copy_longs(StubGenStubId::copy_byte_f_id, copy_f, c_rarg0, c_rarg1, t1); generate_copy_longs(StubGenStubId::copy_byte_b_id, copy_b, c_rarg0, c_rarg1, t1); + address ucm_common_error_exit = generate_unsafecopy_common_error_exit(); + UnsafeMemoryAccess::set_common_exit_stub_pc(ucm_common_error_exit); + StubRoutines::riscv::_zero_blocks = generate_zero_blocks(); //*** jbyte @@ -2259,6 +2377,8 @@ class StubGenerator: public StubCodeGenerator { StubRoutines::_arrayof_jbyte_fill = generate_fill(StubGenStubId::arrayof_jbyte_fill_id); StubRoutines::_arrayof_jshort_fill = generate_fill(StubGenStubId::arrayof_jshort_fill_id); StubRoutines::_arrayof_jint_fill = generate_fill(StubGenStubId::arrayof_jint_fill_id); + + StubRoutines::_unsafe_setmemory = generate_unsafe_setmemory(); } void generate_aes_loadkeys(const Register &key, VectorRegister *working_vregs, int rounds) { @@ -6458,58 +6578,6 @@ static const int64_t right_3_bits = right_n_bits(3); return start; } - void generate_vector_math_stubs() { - if (!UseRVV) { - log_info(library)("vector is not supported, skip loading vector math (sleef) library!"); - return; - } - - // Get native vector math stub routine addresses - void* libsleef = nullptr; - char ebuf[1024]; - char dll_name[JVM_MAXPATHLEN]; - if (os::dll_locate_lib(dll_name, sizeof(dll_name), Arguments::get_dll_dir(), "sleef")) { - libsleef = os::dll_load(dll_name, ebuf, sizeof ebuf); - } - if (libsleef == nullptr) { - log_info(library)("Failed to load native vector math (sleef) library, %s!", ebuf); - return; - } - - // Method naming convention - // All the methods are named as _ - // - // Where: - // is the operation name, e.g. sin, cos - // is to indicate float/double - // "fx/dx" for vector float/double operation - // is the precision level - // "u10/u05" represents 1.0/0.5 ULP error bounds - // We use "u10" for all operations by default - // But for those functions do not have u10 support, we use "u05" instead - // rvv, indicates riscv vector extension - // - // e.g. sinfx_u10rvv is the method for computing vector float sin using rvv instructions - // - log_info(library)("Loaded library %s, handle " INTPTR_FORMAT, JNI_LIB_PREFIX "sleef" JNI_LIB_SUFFIX, p2i(libsleef)); - - for (int op = 0; op < VectorSupport::NUM_VECTOR_OP_MATH; op++) { - int vop = VectorSupport::VECTOR_OP_MATH_START + op; - if (vop == VectorSupport::VECTOR_OP_TANH) { // skip tanh because of performance regression - continue; - } - - // The native library does not support u10 level of "hypot". - const char* ulf = (vop == VectorSupport::VECTOR_OP_HYPOT) ? "u05" : "u10"; - - snprintf(ebuf, sizeof(ebuf), "%sfx_%srvv", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_SCALABLE][op] = (address)os::dll_lookup(libsleef, ebuf); - - snprintf(ebuf, sizeof(ebuf), "%sdx_%srvv", VectorSupport::mathname[op], ulf); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_SCALABLE][op] = (address)os::dll_lookup(libsleef, ebuf); - } - } - #endif // COMPILER2 /** @@ -6741,8 +6809,6 @@ static const int64_t right_3_bits = right_n_bits(3); generate_string_indexof_stubs(); - generate_vector_math_stubs(); - #endif // COMPILER2 } diff --git a/src/hotspot/cpu/riscv/templateInterpreterGenerator_riscv.cpp b/src/hotspot/cpu/riscv/templateInterpreterGenerator_riscv.cpp index 72e1180164b..b8de3547c83 100644 --- a/src/hotspot/cpu/riscv/templateInterpreterGenerator_riscv.cpp +++ b/src/hotspot/cpu/riscv/templateInterpreterGenerator_riscv.cpp @@ -765,6 +765,10 @@ void TemplateInterpreterGenerator::lock_method() { // xcpool: cp cache // stack_pointer: previous sp void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { + // Save ConstMethod* in x15_const_method for later use to avoid loading multiple times + Register x15_const_method = x15; + __ ld(x15_const_method, Address(xmethod, Method::const_offset())); + // initialize fixed part of activation frame if (native_call) { __ subi(esp, sp, 14 * wordSize); @@ -775,8 +779,7 @@ void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { __ sd(zr, Address(sp, 12 * wordSize)); } else { __ subi(esp, sp, 12 * wordSize); - __ ld(t0, Address(xmethod, Method::const_offset())); // get ConstMethod - __ add(xbcp, t0, in_bytes(ConstMethod::codes_offset())); // get codebase + __ add(xbcp, x15_const_method, in_bytes(ConstMethod::codes_offset())); // get codebase __ subi(sp, sp, 12 * wordSize); } __ sd(xbcp, Address(sp, wordSize)); @@ -798,9 +801,10 @@ void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { __ sd(fp, Address(sp, 10 * wordSize)); __ la(fp, Address(sp, 12 * wordSize)); // include ra & fp - __ ld(xcpool, Address(xmethod, Method::const_offset())); - __ ld(xcpool, Address(xcpool, ConstMethod::constants_offset())); - __ ld(xcpool, Address(xcpool, ConstantPool::cache_offset())); + // Save ConstantPool* in x28_constants for later use to avoid loading multiple times + Register x28_constants = x28; + __ ld(x28_constants, Address(x15_const_method, ConstMethod::constants_offset())); + __ ld(xcpool, Address(x28_constants, ConstantPool::cache_offset())); __ sd(xcpool, Address(sp, 3 * wordSize)); __ sub(t0, xlocals, fp); __ srai(t0, t0, Interpreter::logStackElementSize); // t0 = xlocals - fp(); @@ -812,13 +816,15 @@ void TemplateInterpreterGenerator::generate_fixed_frame(bool native_call) { __ sd(x19_sender_sp, Address(sp, 9 * wordSize)); __ sd(zr, Address(sp, 8 * wordSize)); - // Get mirror and store it in the frame as GC root for this Method* - __ load_mirror(t2, xmethod, x15, t1); + // Get mirror, Resolve ConstantPool* -> InstanceKlass* -> Java mirror + // and store it in the frame as GC root for this Method* + __ ld(t2, Address(x28_constants, ConstantPool::pool_holder_offset())); + __ ld(t2, Address(t2, in_bytes(Klass::java_mirror_offset()))); + __ resolve_oop_handle(t2, t0, t1); __ sd(t2, Address(sp, 4 * wordSize)); if (!native_call) { - __ ld(t0, Address(xmethod, Method::const_offset())); - __ lhu(t0, Address(t0, ConstMethod::max_stack_offset())); + __ lhu(t0, Address(x15_const_method, ConstMethod::max_stack_offset())); __ add(t0, t0, MAX2(3, Method::extra_stack_entries())); __ slli(t0, t0, 3); __ sub(t0, sp, t0); @@ -1372,6 +1378,31 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) { __ bind(L); } + #if INCLUDE_JFR + __ enter_jfr_critical_section(); + + // This poll test is to uphold the invariant that a JFR sampled frame + // must not return to its caller without a prior safepoint poll check. + // The earlier poll check in this routine is insufficient for this purpose + // because the thread has transitioned back to Java. + + Label slow_path; + Label fast_path; + __ safepoint_poll(slow_path, true /* at_return */, false /* acquire */, false /* in_nmethod */); + __ j(fast_path); + + __ bind(slow_path); + __ push(dtos); + __ push(ltos); + __ set_last_Java_frame(esp, fp, __ pc(), t0); + __ super_call_VM_leaf(CAST_FROM_FN_PTR(address, InterpreterRuntime::at_unwind), xthread); + __ reset_last_Java_frame(true); + __ pop(ltos); + __ pop(dtos); + __ bind(fast_path); + +#endif // INCLUDE_JFR + // jvmti support // Note: This must happen _after_ handling/throwing any exceptions since // the exception handler code notifies the runtime of method exits @@ -1385,10 +1416,13 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) { __ jalr(result_handler); // remove activation - __ ld(esp, Address(fp, frame::interpreter_frame_sender_sp_offset * wordSize)); // get sender sp + // get sender sp + __ ld(esp, Address(fp, frame::interpreter_frame_sender_sp_offset * wordSize)); // remove frame anchor __ leave(); + JFR_ONLY(__ leave_jfr_critical_section();) + // restore sender sp __ mv(sp, esp); @@ -1612,6 +1646,7 @@ void TemplateInterpreterGenerator::generate_throw_exception() { Interpreter::_remove_activation_preserving_args_entry = __ pc(); __ empty_expression_stack(); + __ restore_bcp(); // We could have returned from deoptimizing this frame, so restore rbcp. // Set the popframe_processing bit in pending_popframe_condition // indicating that we are currently handling popframe, so that // call_VMs that may happen later do not trigger new popframe diff --git a/src/hotspot/cpu/riscv/templateTable_riscv.cpp b/src/hotspot/cpu/riscv/templateTable_riscv.cpp index a035326be01..f6bf1e79f92 100644 --- a/src/hotspot/cpu/riscv/templateTable_riscv.cpp +++ b/src/hotspot/cpu/riscv/templateTable_riscv.cpp @@ -125,24 +125,6 @@ static inline Address at_tos_p5() { return Address(esp, Interpreter::expr_offset_in_bytes(5)); } -// Miscellaneous helper routines -// Store an oop (or null) at the Address described by obj. -// If val == noreg this means store a null -static void do_oop_store(InterpreterMacroAssembler* _masm, - Address dst, - Register val, - DecoratorSet decorators) { - assert(val == noreg || val == x10, "parameter is just for looks"); - __ store_heap_oop(dst, val, x28, x29, x13, decorators); -} - -static void do_oop_load(InterpreterMacroAssembler* _masm, - Address src, - Register dst, - DecoratorSet decorators) { - __ load_heap_oop(dst, src, x28, x29, decorators); -} - Address TemplateTable::at_bcp(int offset) { assert(_desc->uses_bcp(), "inconsistent uses_bcp information"); return Address(xbcp, offset); @@ -787,7 +769,7 @@ void TemplateTable::aaload() { index_check(x10, x11); // leaves index in x11 __ addi(x11, x11, arrayOopDesc::base_offset_in_bytes(T_OBJECT) >> LogBytesPerHeapOop); __ shadd(x10, x11, x10, t0, LogBytesPerHeapOop); - do_oop_load(_masm, Address(x10), x10, IS_ARRAY); + __ load_heap_oop(x10, Address(x10), x28, x29, IS_ARRAY); } void TemplateTable::baload() { @@ -1099,7 +1081,7 @@ void TemplateTable::aastore() { // Get the value we will store __ ld(x10, at_tos()); // Now store using the appropriate barrier - do_oop_store(_masm, element_address, x10, IS_ARRAY); + __ store_heap_oop(element_address, x10, x28, x29, x13, IS_ARRAY); __ j(done); // Have a null in x10, x13=array, x12=index. Store null at ary[idx] @@ -1107,7 +1089,7 @@ void TemplateTable::aastore() { __ profile_null_seen(x12); // Store a null - do_oop_store(_masm, element_address, noreg, IS_ARRAY); + __ store_heap_oop(element_address, noreg, x28, x29, x13, IS_ARRAY); // Pop stack arguments __ bind(done); @@ -1757,6 +1739,8 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) { __ mv(x9, x10); // save the nmethod + JFR_ONLY(__ enter_jfr_critical_section();) + call_VM(noreg, CAST_FROM_FN_PTR(address, SharedRuntime::OSR_migration_begin)); // x10 is OSR buffer, move it to expected parameter location @@ -1765,9 +1749,12 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) { // remove activation // get sender esp __ ld(esp, - Address(fp, frame::interpreter_frame_sender_sp_offset * wordSize)); + Address(fp, frame::interpreter_frame_sender_sp_offset * wordSize)); // remove frame anchor __ leave(); + + JFR_ONLY(__ leave_jfr_critical_section();) + // Ensure compiled code always sees stack at proper alignment __ andi(sp, esp, -16); @@ -2560,7 +2547,7 @@ void TemplateTable::getfield_or_static(int byte_no, bool is_static, RewriteContr __ subi(t0, tos_state, (u1)atos); __ bnez(t0, notObj); // atos - do_oop_load(_masm, field, x10, IN_HEAP); + __ load_heap_oop(x10, field, x28, x29, IN_HEAP); __ push(atos); if (rc == may_rewrite) { patch_bytecode(Bytecodes::_fast_agetfield, bc, x11); @@ -2804,7 +2791,7 @@ void TemplateTable::putfield_or_static(int byte_no, bool is_static, RewriteContr __ add(off, obj, off); // if static, obj from cache, else obj from stack. const Address field(off, 0); // Store into the field - do_oop_store(_masm, field, x10, IN_HEAP); + __ store_heap_oop(field, x10, x28, x29, x13, IN_HEAP); if (rc == may_rewrite) { patch_bytecode(Bytecodes::_fast_aputfield, bc, x11, true, byte_no); } @@ -3046,10 +3033,10 @@ void TemplateTable::fast_storefield(TosState state) { __ add(x11, x12, x11); const Address field(x11, 0); - // access field + // access field, must not clobber x13 - flags switch (bytecode()) { case Bytecodes::_fast_aputfield: - do_oop_store(_masm, field, x10, IN_HEAP); + __ store_heap_oop(field, x10, x28, x29, x15, IN_HEAP); break; case Bytecodes::_fast_lputfield: __ access_store_at(T_LONG, IN_HEAP, field, x10, noreg, noreg, noreg); @@ -3128,7 +3115,7 @@ void TemplateTable::fast_accessfield(TosState state) { // access field switch (bytecode()) { case Bytecodes::_fast_agetfield: - do_oop_load(_masm, field, x10, IN_HEAP); + __ load_heap_oop(x10, field, x28, x29, IN_HEAP); __ verify_oop(x10); break; case Bytecodes::_fast_lgetfield: @@ -3186,7 +3173,7 @@ void TemplateTable::fast_xaccess(TosState state) { break; case atos: __ add(x10, x10, x11); - do_oop_load(_masm, Address(x10, 0), x10, IN_HEAP); + __ load_heap_oop(x10, Address(x10, 0), x28, x29, IN_HEAP); __ verify_oop(x10); break; case ftos: diff --git a/src/hotspot/cpu/riscv/vm_version_riscv.cpp b/src/hotspot/cpu/riscv/vm_version_riscv.cpp index a0de9d767bf..eca1bb83ab6 100644 --- a/src/hotspot/cpu/riscv/vm_version_riscv.cpp +++ b/src/hotspot/cpu/riscv/vm_version_riscv.cpp @@ -240,22 +240,20 @@ void VM_Version::common_initialize() { } // UseZvfh (depends on RVV) - if (UseZvfh && !UseRVV) { - warning("Cannot enable UseZvfh on cpu without RVV support."); - FLAG_SET_DEFAULT(UseZvfh, false); + if (UseZvfh) { + if (!UseRVV) { + warning("Cannot enable UseZvfh on cpu without RVV support."); + FLAG_SET_DEFAULT(UseZvfh, false); + } + if (!UseZfh) { + warning("Cannot enable UseZvfh on cpu without Zfh support."); + FLAG_SET_DEFAULT(UseZvfh, false); + } } } #ifdef COMPILER2 void VM_Version::c2_initialize() { - if (UseCMoveUnconditionally) { - FLAG_SET_DEFAULT(UseCMoveUnconditionally, false); - } - - if (ConditionalMoveLimit > 0) { - FLAG_SET_DEFAULT(ConditionalMoveLimit, 0); - } - if (!UseRVV) { FLAG_SET_DEFAULT(MaxVectorSize, 0); } else { @@ -476,7 +474,7 @@ void VM_Version::initialize_cpu_information(void) { _no_of_threads = _no_of_cores; _no_of_sockets = _no_of_cores; snprintf(_cpu_name, CPU_TYPE_DESC_BUF_SIZE - 1, "RISCV64"); - snprintf(_cpu_desc, CPU_DETAILED_DESC_BUF_SIZE, "RISCV64 %s", features_string()); + snprintf(_cpu_desc, CPU_DETAILED_DESC_BUF_SIZE, "RISCV64 %s", cpu_info_string()); _initialized = true; } diff --git a/src/hotspot/cpu/riscv/vm_version_riscv.hpp b/src/hotspot/cpu/riscv/vm_version_riscv.hpp index 4214d6c53dc..a0a42fb5463 100644 --- a/src/hotspot/cpu/riscv/vm_version_riscv.hpp +++ b/src/hotspot/cpu/riscv/vm_version_riscv.hpp @@ -221,13 +221,13 @@ class VM_Version : public Abstract_VM_Version { FLAG_SET_DEFAULT(UseExtension, true); \ } \ - // https://github.com/riscv/riscv-profiles/blob/main/profiles.adoc#rva20-profiles + // https://github.com/riscv/riscv-profiles/blob/main/src/profiles.adoc#rva20-profiles #define RV_USE_RVA20U64 \ RV_ENABLE_EXTENSION(UseRVC) \ static void useRVA20U64Profile(); - // https://github.com/riscv/riscv-profiles/blob/main/profiles.adoc#rva22-profiles + // https://github.com/riscv/riscv-profiles/blob/main/src/profiles.adoc#rva22-profiles #define RV_USE_RVA22U64 \ RV_ENABLE_EXTENSION(UseRVC) \ RV_ENABLE_EXTENSION(UseZba) \ @@ -241,7 +241,7 @@ class VM_Version : public Abstract_VM_Version { static void useRVA22U64Profile(); - // https://github.com/riscv/riscv-profiles/blob/main/rva23-profile.adoc#rva23u64-profile + // https://github.com/riscv/riscv-profiles/blob/main/src/rva23-profile.adoc#rva23u64-profile #define RV_USE_RVA23U64 \ RV_ENABLE_EXTENSION(UseRVC) \ RV_ENABLE_EXTENSION(UseRVV) \ diff --git a/src/hotspot/cpu/s390/abstractInterpreter_s390.cpp b/src/hotspot/cpu/s390/abstractInterpreter_s390.cpp index e815542a51e..96990f0ce94 100644 --- a/src/hotspot/cpu/s390/abstractInterpreter_s390.cpp +++ b/src/hotspot/cpu/s390/abstractInterpreter_s390.cpp @@ -176,7 +176,7 @@ void AbstractInterpreter::layout_activation(Method* method, intptr_t* monitor_base = (intptr_t*)((address)interpreter_frame->fp() - frame::z_ijava_state_size); intptr_t* monitor = monitor_base - (moncount * frame::interpreter_frame_monitor_size()); intptr_t* operand_stack_base = monitor; - intptr_t* tos = operand_stack_base - tempcount - popframe_extra_args; + intptr_t* esp = operand_stack_base - tempcount - popframe_extra_args - 1; intptr_t* top_frame_sp = operand_stack_base - method->max_stack() - frame::z_top_ijava_frame_abi_size / Interpreter::stackElementSize; intptr_t* sender_sp; @@ -206,7 +206,7 @@ void AbstractInterpreter::layout_activation(Method* method, interpreter_frame->interpreter_frame_set_locals(locals_base); interpreter_frame->interpreter_frame_set_monitor_end((BasicObjectLock *)monitor); *interpreter_frame->interpreter_frame_cache_addr() = method->constants()->cache(); - interpreter_frame->interpreter_frame_set_tos_address(tos); + interpreter_frame->interpreter_frame_set_esp(esp); if (!is_bottom_frame) { interpreter_frame->interpreter_frame_set_sender_sp(sender_sp); } diff --git a/src/hotspot/cpu/s390/c1_CodeStubs_s390.cpp b/src/hotspot/cpu/s390/c1_CodeStubs_s390.cpp index c858a4b8cb1..430928a66ed 100644 --- a/src/hotspot/cpu/s390/c1_CodeStubs_s390.cpp +++ b/src/hotspot/cpu/s390/c1_CodeStubs_s390.cpp @@ -52,7 +52,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { CHECK_BAILOUT(); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); return; } @@ -74,7 +74,7 @@ void RangeCheckStub::emit_code(LIR_Assembler* ce) { CHECK_BAILOUT(); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } PredicateFailedStub::PredicateFailedStub(CodeEmitInfo* info) { @@ -88,7 +88,7 @@ void PredicateFailedStub::emit_code(LIR_Assembler* ce) { CHECK_BAILOUT(); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void CounterOverflowStub::emit_code(LIR_Assembler* ce) { @@ -116,7 +116,7 @@ void DivByZeroStub::emit_code(LIR_Assembler* ce) { ce->emit_call_c(Runtime1::entry_for (C1StubId::throw_div0_exception_id)); CHECK_BAILOUT(); ce->add_call_info_here(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } void ImplicitNullCheckStub::emit_code(LIR_Assembler* ce) { @@ -134,7 +134,7 @@ void ImplicitNullCheckStub::emit_code(LIR_Assembler* ce) { CHECK_BAILOUT(); ce->add_call_info_here(_info); ce->verify_oop_map(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } // Note: pass object in Z_R1_scratch @@ -147,7 +147,7 @@ void SimpleExceptionStub::emit_code(LIR_Assembler* ce) { ce->emit_call_c(a); CHECK_BAILOUT(); ce->add_call_info_here(_info); - debug_only(__ should_not_reach_here()); + DEBUG_ONLY(__ should_not_reach_here()); } NewInstanceStub::NewInstanceStub(LIR_Opr klass_reg, LIR_Opr result, ciInstanceKlass* klass, CodeEmitInfo* info, C1StubId stub_id) { diff --git a/src/hotspot/cpu/s390/c1_FrameMap_s390.cpp b/src/hotspot/cpu/s390/c1_FrameMap_s390.cpp index 9fa6da8341f..ddba445154a 100644 --- a/src/hotspot/cpu/s390/c1_FrameMap_s390.cpp +++ b/src/hotspot/cpu/s390/c1_FrameMap_s390.cpp @@ -144,13 +144,13 @@ LIR_Opr FrameMap::_caller_save_fpu_regs[] = {}; // c1 rnr -> FloatRegister FloatRegister FrameMap::nr2floatreg (int rnr) { assert(_init_done, "tables not initialized"); - debug_only(fpu_range_check(rnr);) + DEBUG_ONLY(fpu_range_check(rnr);) return _fpu_rnr2reg[rnr]; } void FrameMap::map_float_register(int rnr, FloatRegister reg) { - debug_only(fpu_range_check(rnr);) - debug_only(fpu_range_check(reg->encoding());) + DEBUG_ONLY(fpu_range_check(rnr);) + DEBUG_ONLY(fpu_range_check(reg->encoding());) _fpu_rnr2reg[rnr] = reg; // mapping c1 regnr. -> FloatRegister _fpu_reg2rnr[reg->encoding()] = rnr; // mapping assembler encoding -> c1 regnr. } diff --git a/src/hotspot/cpu/s390/c1_FrameMap_s390.hpp b/src/hotspot/cpu/s390/c1_FrameMap_s390.hpp index 66ccc8de876..721995f41fe 100644 --- a/src/hotspot/cpu/s390/c1_FrameMap_s390.hpp +++ b/src/hotspot/cpu/s390/c1_FrameMap_s390.hpp @@ -107,7 +107,7 @@ static int fpu_reg2rnr (FloatRegister reg) { assert(_init_done, "tables not initialized"); int c1rnr = _fpu_reg2rnr[reg->encoding()]; - debug_only(fpu_range_check(c1rnr);) + DEBUG_ONLY(fpu_range_check(c1rnr);) return c1rnr; } diff --git a/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp b/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp index 5691a2055b3..0e873250dca 100644 --- a/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp +++ b/src/hotspot/cpu/s390/c1_MacroAssembler_s390.cpp @@ -69,17 +69,18 @@ void C1_MacroAssembler::lock_object(Register Rmark, Register Roop, Register Rbox // Save object being locked into the BasicObjectLock... z_stg(Roop, Address(Rbox, BasicObjectLock::obj_offset())); - if (DiagnoseSyncOnValueBasedClasses != 0) { - load_klass(tmp, Roop); - z_tm(Address(tmp, Klass::misc_flags_offset()), KlassFlags::_misc_is_value_based_class); - branch_optimized(Assembler::bcondAllOne, slow_case); - } - assert(LockingMode != LM_MONITOR, "LM_MONITOR is already handled, by emit_lock()"); if (LockingMode == LM_LIGHTWEIGHT) { lightweight_lock(Rbox, Roop, Rmark, tmp, slow_case); } else if (LockingMode == LM_LEGACY) { + + if (DiagnoseSyncOnValueBasedClasses != 0) { + load_klass(tmp, Roop); + z_tm(Address(tmp, Klass::misc_flags_offset()), KlassFlags::_misc_is_value_based_class); + branch_optimized(Assembler::bcondAllOne, slow_case); + } + NearLabel done; // Load object header. diff --git a/src/hotspot/cpu/s390/frame_s390.cpp b/src/hotspot/cpu/s390/frame_s390.cpp index 01ed22c7d86..b602d0adce5 100644 --- a/src/hotspot/cpu/s390/frame_s390.cpp +++ b/src/hotspot/cpu/s390/frame_s390.cpp @@ -185,7 +185,8 @@ bool frame::is_interpreted_frame() const { void frame::interpreter_frame_set_locals(intptr_t* locs) { assert(is_interpreted_frame(), "interpreted frame expected"); - ijava_state_unchecked()->locals = (uint64_t)locs; + // set relativized locals + *addr_at(_z_ijava_idx(locals)) = (intptr_t) (locs - fp()); } // sender_sp @@ -340,7 +341,7 @@ bool frame::is_interpreted_frame_valid(JavaThread* thread) const { if (MetaspaceObj::is_valid(cp) == false) return false; // validate locals - address locals = (address)(ijava_state_unchecked()->locals); + address locals = (address)interpreter_frame_locals(); return thread->is_in_stack_range_incl(locals, (address)fp()); } diff --git a/src/hotspot/cpu/s390/frame_s390.hpp b/src/hotspot/cpu/s390/frame_s390.hpp index 3a6b3f33a55..ad754706367 100644 --- a/src/hotspot/cpu/s390/frame_s390.hpp +++ b/src/hotspot/cpu/s390/frame_s390.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2016, 2025, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2016, 2024 SAP SE. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * @@ -330,6 +330,10 @@ #define _z_ijava_state_neg(_component) \ (int) (-frame::z_ijava_state_size + offset_of(frame::z_ijava_state, _component)) +// Frame slot index relative to fp +#define _z_ijava_idx(_component) \ + (_z_ijava_state_neg(_component) >> LogBytesPerWord) + // ENTRY_FRAME struct z_entry_frame_locals { @@ -406,7 +410,7 @@ // C2I adapter frames: // - // STACK (interpreted called from compiled, on entry to frame manager): + // STACK (interpreted called from compiled, on entry to template interpreter): // // [TOP_C2I_FRAME] // [JIT_FRAME] @@ -471,6 +475,7 @@ public: // To be used, if sp was not extended to match callee's calling convention. inline frame(intptr_t* sp, address pc, intptr_t* unextended_sp = nullptr, intptr_t* fp = nullptr, CodeBlob* cb = nullptr); + inline frame(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc, CodeBlob* cb, const ImmutableOopMap* oop_map = nullptr); // Access frame via stack pointer. inline intptr_t* sp_addr_at(int index) const { return &sp()[index]; } @@ -494,14 +499,12 @@ inline z_ijava_state* ijava_state() const; - // Where z_ijava_state.monitors is saved. - inline BasicObjectLock** interpreter_frame_monitors_addr() const; - // Where z_ijava_state.esp is saved. - inline intptr_t** interpreter_frame_esp_addr() const; - public: + + inline intptr_t* interpreter_frame_esp() const; + // Where z_ijava_state.esp is saved. + inline void interpreter_frame_set_esp(intptr_t* esp); inline intptr_t* interpreter_frame_top_frame_sp(); - inline void interpreter_frame_set_tos_address(intptr_t* x); inline void interpreter_frame_set_top_frame_sp(intptr_t* top_frame_sp); inline void interpreter_frame_set_sender_sp(intptr_t* sender_sp); #ifdef ASSERT @@ -513,6 +516,8 @@ // Next two functions read and write z_ijava_state.monitors. private: inline BasicObjectLock* interpreter_frame_monitors() const; + + // Where z_ijava_state.monitors is saved. inline void interpreter_frame_set_monitors(BasicObjectLock* monitors); public: diff --git a/src/hotspot/cpu/s390/frame_s390.inline.hpp b/src/hotspot/cpu/s390/frame_s390.inline.hpp index d29106cfc40..dea0e72581f 100644 --- a/src/hotspot/cpu/s390/frame_s390.inline.hpp +++ b/src/hotspot/cpu/s390/frame_s390.inline.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2016, 2024, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2016, 2025, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2016, 2024 SAP SE. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * @@ -87,6 +87,11 @@ inline frame::frame(intptr_t* sp, address pc, intptr_t* unextended_sp, intptr_t* inline frame::frame(intptr_t* sp) : frame(sp, nullptr) {} +inline frame::frame(intptr_t* sp, intptr_t* unextended_sp, intptr_t* fp, address pc, CodeBlob* cb, const ImmutableOopMap* oop_map) + :_sp(sp), _pc(pc), _cb(cb), _oop_map(oop_map), _on_heap(false), DEBUG_ONLY(_frame_index(-1) COMMA) _unextended_sp(unextended_sp), _fp(fp) { + setup(); +} + // Generic constructor. Used by pns() in debug.cpp only #ifndef PRODUCT inline frame::frame(void* sp, void* pc, void* unextended_sp) @@ -109,16 +114,19 @@ inline frame::z_ijava_state* frame::ijava_state() const { return state; } -inline BasicObjectLock** frame::interpreter_frame_monitors_addr() const { - return (BasicObjectLock**) &(ijava_state()->monitors); -} - // The next two functions read and write z_ijava_state.monitors. inline BasicObjectLock* frame::interpreter_frame_monitors() const { - return *interpreter_frame_monitors_addr(); + BasicObjectLock* result = (BasicObjectLock*) at_relative(_z_ijava_idx(monitors)); + // make sure the pointer points inside the frame + assert(sp() <= (intptr_t*) result, "monitor end should be above the stack pointer"); + assert((intptr_t*) result < fp(), "monitor end should be strictly below the frame pointer: result: " INTPTR_FORMAT " fp: " INTPTR_FORMAT, p2i(result), p2i(fp())); + return result; } + inline void frame::interpreter_frame_set_monitors(BasicObjectLock* monitors) { - *interpreter_frame_monitors_addr() = monitors; + assert(is_interpreted_frame(), "interpreted frame expected"); + // set relativized monitors + ijava_state()->monitors = (intptr_t) ((intptr_t*)monitors - fp()); } // Accessors @@ -180,7 +188,8 @@ inline intptr_t* frame::link_or_null() const { } inline intptr_t* frame::interpreter_frame_locals() const { - return (intptr_t*) (ijava_state()->locals); + intptr_t n = *addr_at(_z_ijava_idx(locals)); + return &fp()[n]; // return relativized locals } inline intptr_t* frame::interpreter_frame_bcp_addr() const { @@ -202,11 +211,14 @@ inline intptr_t* frame::interpreter_frame_expression_stack() const { // Also begin is one past last monitor. inline intptr_t* frame::interpreter_frame_top_frame_sp() { - return (intptr_t*)ijava_state()->top_frame_sp; + intptr_t n = *addr_at(_z_ijava_idx(top_frame_sp)); + return &fp()[n]; // return relativized locals } inline void frame::interpreter_frame_set_top_frame_sp(intptr_t* top_frame_sp) { - ijava_state()->top_frame_sp = (intptr_t) top_frame_sp; + assert(is_interpreted_frame(), "interpreted frame expected"); + // set relativized top_frame_sp + ijava_state()->top_frame_sp = (intptr_t) (top_frame_sp - fp()); } inline void frame::interpreter_frame_set_sender_sp(intptr_t* sender_sp) { @@ -219,18 +231,20 @@ inline void frame::interpreter_frame_set_magic() { } #endif +inline intptr_t* frame::interpreter_frame_esp() const { + return (intptr_t*) at_relative(_z_ijava_idx(esp)); +} + // Where z_ijava_state.esp is saved. -inline intptr_t** frame::interpreter_frame_esp_addr() const { - return (intptr_t**) &(ijava_state()->esp); +inline void frame::interpreter_frame_set_esp(intptr_t* esp) { + assert(is_interpreted_frame(), "interpreted frame expected"); + // set relativized esp + ijava_state()->esp = (intptr_t) (esp - fp()); } // top of expression stack (lowest address) inline intptr_t* frame::interpreter_frame_tos_address() const { - return *interpreter_frame_esp_addr() + 1; -} - -inline void frame::interpreter_frame_set_tos_address(intptr_t* x) { - *interpreter_frame_esp_addr() = x - 1; + return interpreter_frame_esp() + Interpreter::stackElementWords; } // Stack slot needed for native calls and GC. @@ -362,4 +376,42 @@ void frame::update_map_with_saved_link(RegisterMapT* map, intptr_t** link_addr) Unimplemented(); } +#if INCLUDE_JFR + +// Static helper routines +inline intptr_t* frame::sender_sp(intptr_t* fp) { return fp; } + +// Extract common_abi parts. +inline intptr_t* frame::fp(const intptr_t* sp) { + assert(sp != nullptr, "invariant"); + return reinterpret_cast (bound)); + } else { + mov64(tmp, bound); + cmpq(val, tmp); + } + }; + + if (hi == max_jlong) { + cmp_val(lo); + jccb(Assembler::greaterEqual, succeed); + } else { + if (lo != min_jlong) { + cmp_val(lo); + jccb(Assembler::less, fail); + } + cmp_val(hi); + jccb(Assembler::lessEqual, succeed); + } + + bind(fail); + movl(c_rarg0, idx); + movq(c_rarg1, val); + mov64(c_rarg2, lo); + mov64(c_rarg3, hi); + reconstruct_frame_pointer(rscratch1); + call(RuntimeAddress(CAST_FROM_FN_PTR(address, abort_verify_long_in_range))); + hlt(); + bind(succeed); + BLOCK_COMMENT("} // CastLL"); +} + //------------------------------------------------------------------------------------------- // Generic instructions support for use in .ad files C2 code generation @@ -5618,7 +5731,7 @@ void C2_MacroAssembler::vector_compress_expand_avx2(int opcode, XMMRegister dst, // in a permute table row contains either a valid permute index or a -1 (default) // value, this can potentially be used as a blending mask after // compressing/expanding the source vector lanes. - vblendvps(dst, dst, xtmp, permv, vec_enc, false, permv); + vblendvps(dst, dst, xtmp, permv, vec_enc, true, permv); } void C2_MacroAssembler::vector_compress_expand(int opcode, XMMRegister dst, XMMRegister src, KRegister mask, diff --git a/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp b/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp index dd2880d88c3..713eb73d68f 100644 --- a/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp +++ b/src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp @@ -44,6 +44,9 @@ Register t, Register thread); void fast_unlock_lightweight(Register obj, Register reg_rax, Register t, Register thread); + void verify_int_in_range(uint idx, const TypeInt* t, Register val); + void verify_long_in_range(uint idx, const TypeLong* t, Register val, Register tmp); + // Generic instructions support for use in .ad files C2 code generation void vabsnegd(int opcode, XMMRegister dst, XMMRegister src); void vabsnegd(int opcode, XMMRegister dst, XMMRegister src, int vector_len); @@ -574,4 +577,7 @@ void scalar_max_min_fp16(int opcode, XMMRegister dst, XMMRegister src1, XMMRegister src2, KRegister ktmp, XMMRegister xtmp1, XMMRegister xtmp2); + + void reconstruct_frame_pointer(Register rtmp); + #endif // CPU_X86_C2_MACROASSEMBLER_X86_HPP diff --git a/src/hotspot/cpu/x86/compressedKlass_x86.cpp b/src/hotspot/cpu/x86/compressedKlass_x86.cpp index 8a06a7ba3d5..e88b7a3d4e1 100644 --- a/src/hotspot/cpu/x86/compressedKlass_x86.cpp +++ b/src/hotspot/cpu/x86/compressedKlass_x86.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023, Red Hat, Inc. All rights reserved. + * Copyright (c) 2023, 2025, Red Hat, Inc. All rights reserved. * Copyright (c) 2023, 2025, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * @@ -25,6 +25,7 @@ #ifdef _LP64 +#include "memory/metaspace.hpp" #include "oops/compressedKlass.hpp" #include "utilities/globalDefinitions.hpp" @@ -32,15 +33,25 @@ char* CompressedKlassPointers::reserve_address_space_for_compressed_classes(size char* result = nullptr; - // Optimize for unscaled encoding; failing that, for zero-based encoding: - if (optimize_for_zero_base) { - result = reserve_address_space_for_unscaled_encoding(size, aslr); - if (result == nullptr) { - result = reserve_address_space_for_zerobased_encoding(size, aslr); - } - } // end: low-address reservation + assert(CompressedKlassPointers::narrow_klass_pointer_bits() == 32 || + CompressedKlassPointers::narrow_klass_pointer_bits() == 22, "Rethink if we ever use different nKlass bit sizes"); + + // Unconditionally attempting to reserve in lower 4G first makes always sense: + // -CDS -COH: Try to get unscaled mode (zero base, zero shift) + // +CDS -COH: No zero base possible (CDS prevents it); but we still benefit from small base pointers (imm32 movabs) + // -CDS +COH: No zero base possible (22bit nKlass + zero base zero shift = 4MB encoding range, way too small); + // but we still benefit from small base pointers (imm32 movabs) + // +CDS +COH: No zero base possible for multiple reasons (CDS prevents it and encoding range too small); + // but we still benefit from small base pointers (imm32 movabs) + + result = reserve_address_space_below_4G(size, aslr); + + if (result == nullptr && optimize_for_zero_base) { + // Failing that, if we are running without CDS, attempt to allocate below 32G. + // This allows us to use zero-based encoding with a non-zero shift. + result = reserve_address_space_for_zerobased_encoding(size, aslr); + } - // Nothing more to optimize for on x64. If base != 0, we will always emit the full 64-bit immediate. return result; } diff --git a/src/hotspot/cpu/x86/frame_x86.cpp b/src/hotspot/cpu/x86/frame_x86.cpp index a5700134f60..b9a2ef35f10 100644 --- a/src/hotspot/cpu/x86/frame_x86.cpp +++ b/src/hotspot/cpu/x86/frame_x86.cpp @@ -701,7 +701,6 @@ void JavaFrameAnchor::make_walkable() { if (last_Java_sp() == nullptr) return; // already walkable? if (walkable()) return; - vmassert(last_Java_pc() == nullptr, "already walkable"); _last_Java_pc = (address)_last_Java_sp[-1]; vmassert(walkable(), "something went wrong"); } diff --git a/src/hotspot/cpu/x86/frame_x86.inline.hpp b/src/hotspot/cpu/x86/frame_x86.inline.hpp index c74731d0410..afc4ab8767b 100644 --- a/src/hotspot/cpu/x86/frame_x86.inline.hpp +++ b/src/hotspot/cpu/x86/frame_x86.inline.hpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 1997, 2024, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 1997, 2025, Oracle and/or its affiliates. All rights reserved. * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. * * This code is free software; you can redistribute it and/or modify it @@ -35,6 +35,53 @@ // Inline functions for Intel frames: +#if INCLUDE_JFR + +// Static helper routines + +inline address frame::interpreter_bcp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::interpreter_frame_bcp_offset]); +} + +inline address frame::interpreter_return_address(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast(fp[frame::return_addr_offset]); +} + +inline intptr_t* frame::interpreter_sender_sp(const intptr_t* fp) { + assert(fp != nullptr, "invariant"); + return reinterpret_cast _ha_ - // Where: - // ha stands for high accuracy - // is optional to indicate float/double - // Set to f for vector float operation - // Omitted for vector double operation - // is the number of elements in the vector - // 1, 2, 4, 8, 16 - // e.g. 128 bit float vector has 4 float elements - // indicates the avx/sse level: - // z0 is AVX512, l9 is AVX2, e9 is AVX1 and ex is for SSE2 - // e.g. __jsvml_expf16_ha_z0 is the method for computing 16 element vector float exp using AVX 512 insns - // __jsvml_exp8_ha_z0 is the method for computing 8 element vector double exp using AVX 512 insns - - log_info(library)("Loaded library %s, handle " INTPTR_FORMAT, JNI_LIB_PREFIX "jsvml" JNI_LIB_SUFFIX, p2i(libjsvml)); - if (UseAVX > 2) { - for (int op = 0; op < VectorSupport::NUM_VECTOR_OP_MATH; op++) { - int vop = VectorSupport::VECTOR_OP_MATH_START + op; - if ((!VM_Version::supports_avx512dq()) && - (vop == VectorSupport::VECTOR_OP_LOG || vop == VectorSupport::VECTOR_OP_LOG10 || vop == VectorSupport::VECTOR_OP_POW)) { - continue; - } - snprintf(ebuf, sizeof(ebuf), "__jsvml_%sf16_ha_z0", VectorSupport::mathname[op]); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_512][op] = (address)os::dll_lookup(libjsvml, ebuf); - - snprintf(ebuf, sizeof(ebuf), "__jsvml_%s8_ha_z0", VectorSupport::mathname[op]); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_512][op] = (address)os::dll_lookup(libjsvml, ebuf); - } - } - const char* avx_sse_str = (UseAVX >= 2) ? "l9" : ((UseAVX == 1) ? "e9" : "ex"); - for (int op = 0; op < VectorSupport::NUM_VECTOR_OP_MATH; op++) { - int vop = VectorSupport::VECTOR_OP_MATH_START + op; - if (vop == VectorSupport::VECTOR_OP_POW) { - continue; - } - snprintf(ebuf, sizeof(ebuf), "__jsvml_%sf4_ha_%s", VectorSupport::mathname[op], avx_sse_str); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_64][op] = (address)os::dll_lookup(libjsvml, ebuf); - - snprintf(ebuf, sizeof(ebuf), "__jsvml_%sf4_ha_%s", VectorSupport::mathname[op], avx_sse_str); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_128][op] = (address)os::dll_lookup(libjsvml, ebuf); - - snprintf(ebuf, sizeof(ebuf), "__jsvml_%sf8_ha_%s", VectorSupport::mathname[op], avx_sse_str); - StubRoutines::_vector_f_math[VectorSupport::VEC_SIZE_256][op] = (address)os::dll_lookup(libjsvml, ebuf); - - snprintf(ebuf, sizeof(ebuf), "__jsvml_%s1_ha_%s", VectorSupport::mathname[op], avx_sse_str); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_64][op] = (address)os::dll_lookup(libjsvml, ebuf); - - snprintf(ebuf, sizeof(ebuf), "__jsvml_%s2_ha_%s", VectorSupport::mathname[op], avx_sse_str); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_128][op] = (address)os::dll_lookup(libjsvml, ebuf); - - snprintf(ebuf, sizeof(ebuf), "__jsvml_%s4_ha_%s", VectorSupport::mathname[op], avx_sse_str); - StubRoutines::_vector_d_math[VectorSupport::VEC_SIZE_256][op] = (address)os::dll_lookup(libjsvml, ebuf); - } - } - #endif // COMPILER2 #endif // COMPILER2_OR_JVMCI } diff --git a/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp b/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp index c08b0168796..2f1e46f3132 100644 --- a/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp +++ b/src/hotspot/cpu/x86/stubGenerator_x86_64.hpp @@ -490,10 +490,13 @@ class StubGenerator: public StubCodeGenerator { // SHA3 stubs void generate_sha3_stubs(); - // Dilithium stubs and helper functions + // Kyber stubs + void generate_kyber_stubs(); + + // Dilithium stubs void generate_dilithium_stubs(); - // BASE64 stubs + // BASE64 stubs address base64_shuffle_addr(); address base64_avx2_shuffle_addr(); address base64_avx2_input_mask_addr(); @@ -553,6 +556,7 @@ class StubGenerator: public StubCodeGenerator { address generate_libmCos(); address generate_libmTan(); address generate_libmTanh(); + address generate_libmCbrt(); address generate_libmExp(); address generate_libmPow(); address generate_libmLog(); diff --git a/src/hotspot/cpu/x86/stubGenerator_x86_64_cbrt.cpp b/src/hotspot/cpu/x86/stubGenerator_x86_64_cbrt.cpp new file mode 100644 index 00000000000..da60a9be276 --- /dev/null +++ b/src/hotspot/cpu/x86/stubGenerator_x86_64_cbrt.cpp @@ -0,0 +1,366 @@ +/* + * Copyright (c) 2025, Intel Corporation. All rights reserved. + * Intel Math Library (LIBM) Source Code + * + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + * + */ + +#include "macroAssembler_x86.hpp" +#include "stubGenerator_x86_64.hpp" + +/******************************************************************************/ +// ALGORITHM DESCRIPTION +// --------------------- +// +// x=2^{3*k+j} * 1.b1 b2 ... b5 b6 ... b52 +// Let r=(x*2^{-3k-j} - 1.b1 b2 ... b5 1)* rcp[b1 b2 ..b5], +// where rcp[b1 b2 .. b5]=1/(1.b1 b2 b3 b4 b5 1) in double precision +// cbrt(2^j * 1. b1 b2 .. b5 1) is approximated as T[j][b1..b5]+D[j][b1..b5] +// (T stores the high 53 bits, D stores the low order bits) +// Result=2^k*T+(2^k*T*r)*P+2^k*D +// where P=p1+p2*r+..+p8*r^7 +// +// Special cases: +// cbrt(NaN) = quiet NaN +// cbrt(+/-INF) = +/-INF +// cbrt(+/-0) = +/-0 +// +/******************************************************************************/ + +ATTRIBUTE_ALIGNED(4) static const juint _SIG_MASK[] = +{ + 0, 1032192 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _EXP_MASK[] = +{ + 0, 3220176896 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _EXP_MSK2[] = +{ + 0, 3220193280 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _EXP_MSK3[] = +{ + 4294967295, 1048575 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _SCALE63[] = +{ + 0, 1138753536 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _ZERON[] = +{ + 0, 2147483648 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _INF[] = +{ + 0, 2146435072 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _NEG_INF[] = +{ + 0, 4293918720 +}; + +ATTRIBUTE_ALIGNED(16) static const juint _coeff_table[] = +{ + 1553778919, 3213899486, 3534952507, 3215266280, 1646371399, + 3214412045, 477218588, 3216798151, 3582521621, 1066628362, + 1007461464, 1068473053, 889629714, 1067378449, 1431655765, + 1070945621 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _rcp_table[] = +{ + 528611360, 3220144632, 2884679527, 3220082993, 1991868891, 3220024928, + 2298714891, 3219970134, 58835168, 3219918343, 3035110223, 3219869313, + 1617585086, 3219822831, 2500867033, 3219778702, 4241943008, 3219736752, + 258732970, 3219696825, 404232216, 3219658776, 2172167368, 3219622476, + 1544257904, 3219587808, 377579543, 3219554664, 1616385542, 3219522945, + 813783277, 3219492562, 3940743189, 3219463431, 2689777499, 3219435478, + 1700977147, 3219408632, 3169102082, 3219382828, 327235604, 3219358008, + 1244336319, 3219334115, 1300311200, 3219311099, 3095471925, 3219288912, + 2166487928, 3219267511, 2913108253, 3219246854, 293672978, 3219226904, + 288737297, 3219207624, 1810275472, 3219188981, 174592167, 3219170945, + 3539053052, 3219153485, 2164392968, 3219136576 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _cbrt_table[] = +{ + 572345495, 1072698681, 1998204467, 1072709382, 3861501553, 1072719872, + 2268192434, 1072730162, 2981979308, 1072740260, 270859143, 1072750176, + 2958651392, 1072759916, 313113243, 1072769490, 919449400, 1072778903, + 2809328903, 1072788162, 2222981587, 1072797274, 2352530781, 1072806244, + 594152517, 1072815078, 1555767199, 1072823780, 4282421314, 1072832355, + 2355578597, 1072840809, 1162590619, 1072849145, 797864051, 1072857367, + 431273680, 1072865479, 2669831148, 1072873484, 733477752, 1072881387, + 4280220604, 1072889189, 801961634, 1072896896, 2915370760, 1072904508, + 1159613482, 1072912030, 2689944798, 1072919463, 1248687822, 1072926811, + 2967951030, 1072934075, 630170432, 1072941259, 3760898254, 1072948363, + 0, 1072955392, 2370273294, 1072962345, 1261754802, 1072972640, + 546334065, 1072986123, 1054893830, 1072999340, 1571187597, 1073012304, + 1107975175, 1073025027, 3606909377, 1073037519, 1113616747, 1073049792, + 4154744632, 1073061853, 3358931423, 1073073713, 4060702372, 1073085379, + 747576176, 1073096860, 3023138255, 1073108161, 1419988548, 1073119291, + 1914185305, 1073130255, 294389948, 1073141060, 3761802570, 1073151710, + 978281566, 1073162213, 823148820, 1073172572, 2420954441, 1073182792, + 3815449908, 1073192878, 2046058587, 1073202835, 1807524753, 1073212666, + 2628681401, 1073222375, 3225667357, 1073231966, 1555307421, 1073241443, + 3454043099, 1073250808, 1208137896, 1073260066, 3659916772, 1073269218, + 1886261264, 1073278269, 3593647839, 1073287220, 3086012205, 1073296075, + 2769796922, 1073304836, 888716057, 1073317807, 2201465623, 1073334794, + 164369365, 1073351447, 3462666733, 1073367780, 2773905457, 1073383810, + 1342879088, 1073399550, 2543933975, 1073415012, 1684477781, 1073430209, + 3532178543, 1073445151, 1147747300, 1073459850, 1928031793, 1073474314, + 2079717015, 1073488553, 4016765315, 1073502575, 3670431139, 1073516389, + 3549227225, 1073530002, 11637607, 1073543422, 588220169, 1073556654, + 2635407503, 1073569705, 2042029317, 1073582582, 1925128962, 1073595290, + 4136375664, 1073607834, 759964600, 1073620221, 4257606771, 1073632453, + 297278907, 1073644538, 3655053093, 1073656477, 2442253172, 1073668277, + 1111876799, 1073679941, 3330973139, 1073691472, 3438879452, 1073702875, + 3671565478, 1073714153, 1317849547, 1073725310, 1642364115, 1073736348 +}; + +ATTRIBUTE_ALIGNED(4) static const juint _D_table[] = +{ + 4050900474, 1014427190, 1157977860, 1016444461, 1374568199, 1017271387, + 2809163288, 1016882676, 3742377377, 1013168191, 3101606597, 1017541672, + 65224358, 1017217597, 2691591250, 1017266643, 4020758549, 1017689313, + 1316310992, 1018030788, 1031537856, 1014090882, 3261395239, 1016413641, + 886424999, 1016313335, 3114776834, 1014195875, 1681120620, 1017825416, + 1329600273, 1016625740, 465474623, 1017097119, 4251633980, 1017169077, + 1986990133, 1017710645, 752958613, 1017159641, 2216216792, 1018020163, + 4282860129, 1015924861, 1557627859, 1016039538, 3889219754, 1018086237, + 3684996408, 1017353275, 723532103, 1017717141, 2951149676, 1012528470, + 831890937, 1017830553, 1031212645, 1017387331, 2741737450, 1017604974, + 2863311531, 1003776682, 4276736099, 1013153088, 4111778382, 1015673686, + 1728065769, 1016413986, 2708718031, 1018078833, 1069335005, 1015291224, + 700037144, 1016482032, 2904566452, 1017226861, 4074156649, 1017622651, + 25019565, 1015245366, 3601952608, 1015771755, 3267129373, 1017904664, + 503203103, 1014921629, 2122011730, 1018027866, 3927295461, 1014189456, + 2790625147, 1016024251, 1330460186, 1016940346, 4033568463, 1015538390, + 3695818227, 1017509621, 257573361, 1017208868, 3227697852, 1017337964, + 234118548, 1017169577, 4009025803, 1017278524, 1948343394, 1017749310, + 678398162, 1018144239, 3083864863, 1016669086, 2415453452, 1017890370, + 175467344, 1017330033, 3197359580, 1010339928, 2071276951, 1015941358, + 268372543, 1016737773, 938132959, 1017389108, 1816750559, 1017337448, + 4119203749, 1017152174, 2578653878, 1013108497, 2470331096, 1014678606, + 123855735, 1016553320, 1265650889, 1014782687, 3414398172, 1017182638, + 1040773369, 1016158401, 3483628886, 1016886550, 4140499405, 1016191425, + 3893477850, 1016964495, 3935319771, 1009634717, 2978982660, 1015027112, + 2452709923, 1017990229, 3190365712, 1015835149, 4237588139, 1015832925, + 2610678389, 1017962711, 2127316774, 1017405770, 824267502, 1017959463, + 2165924042, 1017912225, 2774007076, 1013257418, 4123916326, 1017582284, + 1976417958, 1016959909, 4092806412, 1017711279, 119251817, 1015363631, + 3475418768, 1017675415, 1972580503, 1015470684, 815541017, 1017517969, + 2429917451, 1017397776, 4062888482, 1016749897, 68284153, 1017925678, + 2207779246, 1016320298, 1183466520, 1017408657, 143326427, 1017060403 +}; + +#define __ _masm-> + +address StubGenerator::generate_libmCbrt() { + StubGenStubId stub_id = StubGenStubId::dcbrt_id; + StubCodeMark mark(this, stub_id); + address start = __ pc(); + + Label L_2TAG_PACKET_0_0_1, L_2TAG_PACKET_1_0_1, L_2TAG_PACKET_2_0_1, L_2TAG_PACKET_3_0_1; + Label L_2TAG_PACKET_4_0_1, L_2TAG_PACKET_5_0_1, L_2TAG_PACKET_6_0_1; + Label B1_1, B1_2, B1_4; + + address SIG_MASK = (address)_SIG_MASK; + address EXP_MASK = (address)_EXP_MASK; + address EXP_MSK2 = (address)_EXP_MSK2; + address EXP_MSK3 = (address)_EXP_MSK3; + address SCALE63 = (address)_SCALE63; + address ZERON = (address)_ZERON; + address INF = (address)_INF; + address NEG_INF = (address)_NEG_INF; + address coeff_table = (address)_coeff_table; + address rcp_table = (address)_rcp_table; + address cbrt_table = (address)_cbrt_table; + address D_table = (address)_D_table; + + __ enter(); // required for proper stackwalking of RuntimeStub frame + + __ bind(B1_1); + __ subq(rsp, 24); + __ movsd(Address(rsp), xmm0); + + __ bind(B1_2); + __ movq(xmm7, xmm0); + __ movl(rdx, 524032); + __ movsd(xmm5, ExternalAddress(EXP_MSK3), r11 /*rscratch*/); + __ movsd(xmm3, ExternalAddress(EXP_MSK2), r11 /*rscratch*/); + __ psrlq(xmm7, 44); + __ pextrw(rcx, xmm7, 0); + __ movdl(rax, xmm7); + __ movsd(xmm1, ExternalAddress(EXP_MASK), r11 /*rscratch*/); + __ movsd(xmm2, ExternalAddress(SIG_MASK), r11 /*rscratch*/); + __ andl(rcx, 248); + __ lea(r8, ExternalAddress(rcp_table)); + __ movsd(xmm4, Address(rcx, r8, Address::times_1)); + __ movq(r9, rax); + __ andl(rdx, rax); + __ cmpl(rdx, 0); + __ jcc(Assembler::equal, L_2TAG_PACKET_0_0_1); // Branch only if |x| is denormalized + __ cmpl(rdx, 524032); + __ jcc(Assembler::equal, L_2TAG_PACKET_1_0_1); // Branch only if |x| is INF or NaN + __ shrl(rdx, 8); + __ shrq(r9, 8); + __ andpd(xmm2, xmm0); + __ andpd(xmm0, xmm5); + __ orpd(xmm3, xmm2); + __ orpd(xmm1, xmm0); + __ movapd(xmm5, ExternalAddress(coeff_table), r11 /*rscratch*/); + __ movl(rax, 5462); + __ movapd(xmm6, ExternalAddress(coeff_table + 16), r11 /*rscratch*/); + __ mull(rdx); + __ movq(rdx, r9); + __ andq(r9, 2047); + __ shrl(rax, 14); + __ andl(rdx, 2048); + __ subq(r9, rax); + __ subq(r9, rax); + __ subq(r9, rax); + __ shlq(r9, 8); + __ addl(rax, 682); + __ orl(rax, rdx); + __ movdl(xmm7, rax); + __ addq(rcx, r9); + __ psllq(xmm7, 52); + + __ bind(L_2TAG_PACKET_2_0_1); + __ movapd(xmm2, ExternalAddress(coeff_table + 32), r11 /*rscratch*/); + __ movapd(xmm0, ExternalAddress(coeff_table + 48), r11 /*rscratch*/); + __ subsd(xmm1, xmm3); + __ movq(xmm3, xmm7); + __ lea(r8, ExternalAddress(cbrt_table)); + __ mulsd(xmm7, Address(rcx, r8, Address::times_1)); + __ mulsd(xmm1, xmm4); + __ lea(r8, ExternalAddress(D_table)); + __ mulsd(xmm3, Address(rcx, r8, Address::times_1)); + __ movapd(xmm4, xmm1); + __ unpcklpd(xmm1, xmm1); + __ mulpd(xmm5, xmm1); + __ mulpd(xmm6, xmm1); + __ mulpd(xmm1, xmm1); + __ addpd(xmm2, xmm5); + __ addpd(xmm0, xmm6); + __ mulpd(xmm2, xmm1); + __ mulpd(xmm1, xmm1); + __ mulsd(xmm4, xmm7); + __ addpd(xmm0, xmm2); + __ mulsd(xmm1, xmm0); + __ unpckhpd(xmm0, xmm0); + __ addsd(xmm0, xmm1); + __ mulsd(xmm0, xmm4); + __ addsd(xmm0, xmm3); + __ addsd(xmm0, xmm7); + __ jmp(B1_4); + + __ bind(L_2TAG_PACKET_0_0_1); + __ mulsd(xmm0, ExternalAddress(SCALE63), r11 /*rscratch*/); + __ movq(xmm7, xmm0); + __ movl(rdx, 524032); + __ psrlq(xmm7, 44); + __ pextrw(rcx, xmm7, 0); + __ movdl(rax, xmm7); + __ andl(rcx, 248); + __ lea(r8, ExternalAddress(rcp_table)); + __ movsd(xmm4, Address(rcx, r8, Address::times_1)); + __ movq(r9, rax); + __ andl(rdx, rax); + __ shrl(rdx, 8); + __ shrq(r9, 8); + __ cmpl(rdx, 0); + __ jcc(Assembler::equal, L_2TAG_PACKET_3_0_1); // Branch only if |x| is zero + __ andpd(xmm2, xmm0); + __ andpd(xmm0, xmm5); + __ orpd(xmm3, xmm2); + __ orpd(xmm1, xmm0); + __ movapd(xmm5, ExternalAddress(coeff_table), r11 /*rscratch*/); + __ movl(rax, 5462); + __ movapd(xmm6, ExternalAddress(coeff_table + 16), r11 /*rscratch*/); + __ mull(rdx); + __ movq(rdx, r9); + __ andq(r9, 2047); + __ shrl(rax, 14); + __ andl(rdx, 2048); + __ subq(r9, rax); + __ subq(r9, rax); + __ subq(r9, rax); + __ shlq(r9, 8); + __ addl(rax, 661); + __ orl(rax, rdx); + __ movdl(xmm7, rax); + __ addq(rcx, r9); + __ psllq(xmm7, 52); + __ jmp(L_2TAG_PACKET_2_0_1); + + __ bind(L_2TAG_PACKET_3_0_1); + __ cmpq(r9, 0); + __ jcc(Assembler::notEqual, L_2TAG_PACKET_4_0_1); // Branch only if x is negative zero + __ xorpd(xmm0, xmm0); + __ jmp(B1_4); + + __ bind(L_2TAG_PACKET_4_0_1); + __ movsd(xmm0, ExternalAddress(ZERON), r11 /*rscratch*/); + __ jmp(B1_4); + + __ bind(L_2TAG_PACKET_1_0_1); + __ movl(rax, Address(rsp, 4)); + __ movl(rdx, Address(rsp)); + __ movl(rcx, rax); + __ andl(rcx, 2147483647); + __ cmpl(rcx, 2146435072); + __ jcc(Assembler::above, L_2TAG_PACKET_5_0_1); // Branch only if |x| is NaN + __ cmpl(rdx, 0); + __ jcc(Assembler::notEqual, L_2TAG_PACKET_5_0_1); // Branch only if |x| is NaN + __ cmpl(rax, 2146435072); + __ jcc(Assembler::notEqual, L_2TAG_PACKET_6_0_1); // Branch only if x is negative INF + __ movsd(xmm0, ExternalAddress(INF), r11 /*rscratch*/); + __ jmp(B1_4); + + __ bind(L_2TAG_PACKET_6_0_1); + __ movsd(xmm0, ExternalAddress(NEG_INF), r11 /*rscratch*/); + __ jmp(B1_4); + + __ bind(L_2TAG_PACKET_5_0_1); + __ movsd(xmm0, Address(rsp)); + __ addsd(xmm0, xmm0); + __ movq(Address(rsp, 8), xmm0); + + __ bind(B1_4); + __ addq(rsp, 24); + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ ret(0); + + return start; +} + +#undef __ diff --git a/src/hotspot/cpu/x86/stubGenerator_x86_64_kyber.cpp b/src/hotspot/cpu/x86/stubGenerator_x86_64_kyber.cpp new file mode 100644 index 00000000000..91c005e92de --- /dev/null +++ b/src/hotspot/cpu/x86/stubGenerator_x86_64_kyber.cpp @@ -0,0 +1,952 @@ +/* + * Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved. + * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. + * + * This code is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 only, as + * published by the Free Software Foundation. + * + * This code is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License + * version 2 for more details (a copy is included in the LICENSE file that + * accompanied this code). + * + * You should have received a copy of the GNU General Public License version + * 2 along with this work; if not, write to the Free Software Foundation, + * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA. + * + * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA + * or visit www.oracle.com if you need additional information or have any + * questions. + * + */ + +#include "asm/assembler.hpp" +#include "asm/assembler.inline.hpp" +#include "runtime/stubRoutines.hpp" +#include "macroAssembler_x86.hpp" +#include "stubGenerator_x86_64.hpp" + +#define __ _masm-> + +#define xmm(i) as_XMMRegister(i) + +#ifdef PRODUCT +#define BLOCK_COMMENT(str) /* nothing */ +#else +#define BLOCK_COMMENT(str) __ block_comment(str) +#endif // PRODUCT + +#define BIND(label) bind(label); BLOCK_COMMENT(#label ":") + +// Constants +// +ATTRIBUTE_ALIGNED(64) static const uint16_t kyberAvx512Consts[] = { + 0xF301, 0xF301, 0xF301, 0xF301, // q^-1 mod montR + 0x0D01, 0x0D01, 0x0D01, 0x0D01, // q + 0x4EBF, 0x4EBF, 0x4EBF, 0x4EBF, // Barrett multiplier + 0x0200, 0x0200, 0x0200, 0x0200, //(dim/2)^-1 mod q + 0x0549, 0x0549, 0x0549, 0x0549, // montR^2 mod q + 0x0F00, 0x0F00, 0x0F00, 0x0F00 // mask for kyber12to16 + }; + +static int qInvModROffset = 0; +static int qOffset = 8; +static int barretMultiplierOffset = 16; +static int dimHalfInverseOffset = 24; +static int montRSquareModqOffset = 32; +static int f00Offset = 40; + +static address kyberAvx512ConstsAddr(int offset) { + return ((address) kyberAvx512Consts) + offset; +} + +const Register scratch = r10; + +ATTRIBUTE_ALIGNED(64) static const uint16_t kyberAvx512NttPerms[] = { +// 0 + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3A, 0x3B, 0x3C, 0x3D, 0x3E, 0x3F, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2A, 0x2B, 0x2C, 0x2D, 0x2E, 0x2F, +// 128 + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x28, 0x29, 0x2A, 0x2B, 0x2C, 0x2D, 0x2E, 0x2F, + 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F, + 0x38, 0x39, 0x3A, 0x3B, 0x3C, 0x3D, 0x3E, 0x3F, +// 256 + 0x00, 0x01, 0x02, 0x03, 0x20, 0x21, 0x22, 0x23, + 0x08, 0x09, 0x0A, 0x0B, 0x28, 0x29, 0x2A, 0x2B, + 0x10, 0x11, 0x12, 0x13, 0x30, 0x31, 0x32, 0x33, + 0x18, 0x19, 0x1A, 0x1B, 0x38, 0x39, 0x3A, 0x3B, + 0x04, 0x05, 0x06, 0x07, 0x24, 0x25, 0x26, 0x27, + 0x0C, 0x0D, 0x0E, 0x0F, 0x2C, 0x2D, 0x2E, 0x2F, + 0x14, 0x15, 0x16, 0x17, 0x34, 0x35, 0x36, 0x37, + 0x1C, 0x1D, 0x1E, 0x1F, 0x3C, 0x3D, 0x3E, 0x3F, +// 384 + 0x00, 0x01, 0x20, 0x21, 0x04, 0x05, 0x24, 0x25, + 0x08, 0x09, 0x28, 0x29, 0x0C, 0x0D, 0x2C, 0x2D, + 0x10, 0x11, 0x30, 0x31, 0x14, 0x15, 0x34, 0x35, + 0x18, 0x19, 0x38, 0x39, 0x1C, 0x1D, 0x3C, 0x3D, + 0x02, 0x03, 0x22, 0x23, 0x06, 0x07, 0x26, 0x27, + 0x0A, 0x0B, 0x2A, 0x2B, 0x0E, 0x0F, 0x2E, 0x2F, + 0x12, 0x13, 0x32, 0x33, 0x16, 0x17, 0x36, 0x37, + 0x1A, 0x1B, 0x3A, 0x3B, 0x1E, 0x1F, 0x3E, 0x3F, +// 512 + 0x10, 0x11, 0x30, 0x31, 0x12, 0x13, 0x32, 0x33, + 0x14, 0x15, 0x34, 0x35, 0x16, 0x17, 0x36, 0x37, + 0x18, 0x19, 0x38, 0x39, 0x1A, 0x1B, 0x3A, 0x3B, + 0x1C, 0x1D, 0x3C, 0x3D, 0x1E, 0x1F, 0x3E, 0x3F, + 0x00, 0x01, 0x20, 0x21, 0x02, 0x03, 0x22, 0x23, + 0x04, 0x05, 0x24, 0x25, 0x06, 0x07, 0x26, 0x27, + 0x08, 0x09, 0x28, 0x29, 0x0A, 0x0B, 0x2A, 0x2B, + 0x0C, 0x0D, 0x2C, 0x2D, 0x0E, 0x0F, 0x2E, 0x2F + }; + +static address kyberAvx512NttPermsAddr() { + return (address) kyberAvx512NttPerms; +} + +ATTRIBUTE_ALIGNED(64) static const uint16_t kyberAvx512InverseNttPerms[] = { +// 0 + 0x02, 0x03, 0x06, 0x07, 0x0A, 0x0B, 0x0E, 0x0F, + 0x12, 0x13, 0x16, 0x17, 0x1A, 0x1B, 0x1E, 0x1F, + 0x22, 0x23, 0x26, 0x27, 0x2A, 0x2B, 0x2E, 0x2F, + 0x32, 0x33, 0x36, 0x37, 0x3A, 0x3B, 0x3E, 0x3F, + 0x00, 0x01, 0x04, 0x05, 0x08, 0x09, 0x0C, 0x0D, + 0x10, 0x11, 0x14, 0x15, 0x18, 0x19, 0x1C, 0x1D, + 0x20, 0x21, 0x24, 0x25, 0x28, 0x29, 0x2C, 0x2D, + 0x30, 0x31, 0x34, 0x35, 0x38, 0x39, 0x3C, 0x3D, +// 128 + 0x00, 0x01, 0x20, 0x21, 0x04, 0x05, 0x24, 0x25, + 0x08, 0x09, 0x28, 0x29, 0x0C, 0x0D, 0x2C, 0x2D, + 0x10, 0x11, 0x30, 0x31, 0x14, 0x15, 0x34, 0x35, + 0x18, 0x19, 0x38, 0x39, 0x1C, 0x1D, 0x3C, 0x3D, + 0x02, 0x03, 0x22, 0x23, 0x06, 0x07, 0x26, 0x27, + 0x0A, 0x0B, 0x2A, 0x2B, 0x0E, 0x0F, 0x2E, 0x2F, + 0x12, 0x13, 0x32, 0x33, 0x16, 0x17, 0x36, 0x37, + 0x1A, 0x1B, 0x3A, 0x3B, 0x1E, 0x1F, 0x3E, 0x3F, +// 256 + 0x00, 0x01, 0x02, 0x03, 0x20, 0x21, 0x22, 0x23, + 0x08, 0x09, 0x0A, 0x0B, 0x28, 0x29, 0x2A, 0x2B, + 0x10, 0x11, 0x12, 0x13, 0x30, 0x31, 0x32, 0x33, + 0x18, 0x19, 0x1A, 0x1B, 0x38, 0x39, 0x3A, 0x3B, + 0x04, 0x05, 0x06, 0x07, 0x24, 0x25, 0x26, 0x27, + 0x0C, 0x0D, 0x0E, 0x0F, 0x2C, 0x2D, 0x2E, 0x2F, + 0x14, 0x15, 0x16, 0x17, 0x34, 0x35, 0x36, 0x37, + 0x1C, 0x1D, 0x1E, 0x1F, 0x3C, 0x3D, 0x3E, 0x3F, +// 384 + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x28, 0x29, 0x2A, 0x2B, 0x2C, 0x2D, 0x2E, 0x2F, + 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F, + 0x38, 0x39, 0x3A, 0x3B, 0x3C, 0x3D, 0x3E, 0x3F, +// 512 + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, + 0x18, 0x19, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F, + 0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, + 0x38, 0x39, 0x3A, 0x3B, 0x3C, 0x3D, 0x3E, 0x3F, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, + 0x28, 0x29, 0x2A, 0x2B, 0x2C, 0x2D, 0x2E, 0x2F + }; + +static address kyberAvx512InverseNttPermsAddr() { + return (address) kyberAvx512InverseNttPerms; +} + +ATTRIBUTE_ALIGNED(64) static const uint16_t kyberAvx512_nttMultPerms[] = { + 0x00, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0E, + 0x10, 0x12, 0x14, 0x16, 0x18, 0x1A, 0x1C, 0x1E, + 0x20, 0x22, 0x24, 0x26, 0x28, 0x2A, 0x2C, 0x2E, + 0x30, 0x32, 0x34, 0x36, 0x38, 0x3A, 0x3C, 0x3E, + + 0x01, 0x03, 0x05, 0x07, 0x09, 0x0B, 0x0D, 0x0F, + 0x11, 0x13, 0x15, 0x17, 0x19, 0x1B, 0x1D, 0x1F, + 0x21, 0x23, 0x25, 0x27, 0x29, 0x2B, 0x2D, 0x2F, + 0x31, 0x33, 0x35, 0x37, 0x39, 0x3B, 0x3D, 0x3F, + + 0x00, 0x20, 0x01, 0x21, 0x02, 0x22, 0x03, 0x23, + 0x04, 0x24, 0x05, 0x25, 0x06, 0x26, 0x07, 0x27, + 0x08, 0x28, 0x09, 0x29, 0x0A, 0x2A, 0x0B, 0x2B, + 0x0C, 0x2C, 0x0D, 0x2D, 0x0E, 0x2E, 0x0F, 0x2F, + + 0x10, 0x30, 0x11, 0x31, 0x12, 0x32, 0x13, 0x33, + 0x14, 0x34, 0x15, 0x35, 0x16, 0x36, 0x17, 0x37, + 0x18, 0x38, 0x19, 0x39, 0x1A, 0x3A, 0x1B, 0x3B, + 0x1C, 0x3C, 0x1D, 0x3D, 0x1E, 0x3E, 0x1F, 0x3F + }; + +static address kyberAvx512_nttMultPermsAddr() { + return (address) kyberAvx512_nttMultPerms; +} + + ATTRIBUTE_ALIGNED(64) static const uint16_t kyberAvx512_12To16Perms[] = { +// 0 + 0x00, 0x03, 0x06, 0x09, 0x0C, 0x0F, 0x12, 0x15, + 0x18, 0x1B, 0x1E, 0x21, 0x24, 0x27, 0x2A, 0x2D, + 0x30, 0x33, 0x36, 0x39, 0x3C, 0x3F, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x01, 0x04, 0x07, 0x0A, 0x0D, 0x10, 0x13, 0x16, + 0x19, 0x1C, 0x1F, 0x22, 0x25, 0x28, 0x2B, 0x2E, + 0x31, 0x34, 0x37, 0x3A, 0x3D, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, +// 128 + 0x02, 0x05, 0x08, 0x0B, 0x0E, 0x11, 0x14, 0x17, + 0x1A, 0x1D, 0x20, 0x23, 0x26, 0x29, 0x2C, 0x2F, + 0x32, 0x35, 0x38, 0x3B, 0x3E, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x22, 0x25, + 0x28, 0x2B, 0x2E, 0x31, 0x34, 0x37, 0x3A, 0x3D, +// 256 + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x20, 0x23, 0x26, + 0x29, 0x2C, 0x2F, 0x32, 0x35, 0x38, 0x3B, 0x3E, + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, + 0x10, 0x11, 0x12, 0x13, 0x14, 0x21, 0x24, 0x27, + 0x2A, 0x2D, 0x30, 0x33, 0x36, 0x39, 0x3C, 0x3F, +// 384 + 0x00, 0x20, 0x01, 0x21, 0x02, 0x22, 0x03, 0x23, + 0x04, 0x24, 0x05, 0x25, 0x06, 0x26, 0x07, 0x27, + 0x08, 0x28, 0x09, 0x29, 0x0A, 0x2A, 0x0B, 0x2B, + 0x0C, 0x2C, 0x0D, 0x2D, 0x0E, 0x2E, 0x0F, 0x2F, + 0x10, 0x30, 0x11, 0x31, 0x12, 0x32, 0x13, 0x33, + 0x14, 0x34, 0x15, 0x35, 0x16, 0x36, 0x17, 0x37, + 0x18, 0x38, 0x19, 0x39, 0x1A, 0x3A, 0x1B, 0x3B, + 0x1C, 0x3C, 0x1D, 0x3D, 0x1E, 0x3E, 0x1F, 0x3F + }; + +static address kyberAvx512_12To16PermsAddr() { + return (address) kyberAvx512_12To16Perms; +} + +static void load4regs(int destRegs[], Register address, int offset, + MacroAssembler *_masm) { + for (int i = 0; i < 4; i++) { + __ evmovdquw(xmm(destRegs[i]), Address(address, offset + i * 64), + Assembler::AVX_512bit); + } +} + +// For z = montmul(a,b), z will be between -q and q and congruent +// to a * b * R^-1 mod q, where R > 2 * q, R is a power of 2, +// -R/2 * q <= a * b < R/2 * q. +// (See e.g. Algorithm 3 in https://eprint.iacr.org/2018/039.pdf) +// For the Java code, we use R = 2^20 and for the intrinsic, R = 2^16. +// In our computations, b is always c * R mod q, so the montmul() really +// computes a * c mod q. In the Java code, we use 32-bit numbers for the +// computations, and we use R = 2^20 because that way the a * b numbers +// that occur during all computations stay in the required range. +// For the intrinsics, we use R = 2^16, because this way we can do twice +// as much work in parallel, the only drawback is that we should do some Barrett +// reductions in kyberInverseNtt so that the numbers stay in the required range. +static void montmul(int outputRegs[], int inputRegs1[], int inputRegs2[], + int scratchRegs1[], int scratchRegs2[], MacroAssembler *_masm) { + for (int i = 0; i < 4; i++) { + __ evpmullw(xmm(scratchRegs1[i]), k0, xmm(inputRegs1[i]), + xmm(inputRegs2[i]), false, Assembler::AVX_512bit); + } + for (int i = 0; i < 4; i++) { + __ evpmulhw(xmm(scratchRegs2[i]), k0, xmm(inputRegs1[i]), + xmm(inputRegs2[i]), false, Assembler::AVX_512bit); + } + for (int i = 0; i < 4; i++) { + __ evpmullw(xmm(scratchRegs1[i]), k0, xmm(scratchRegs1[i]), + xmm31, false, Assembler::AVX_512bit); + } + for (int i = 0; i < 4; i++) { + __ evpmulhw(xmm(scratchRegs1[i]), k0, xmm(scratchRegs1[i]), + xmm30, false, Assembler::AVX_512bit); + } + for (int i = 0; i < 4; i++) { + __ evpsubw(xmm(outputRegs[i]), k0, xmm(scratchRegs2[i]), + xmm(scratchRegs1[i]), false, Assembler::AVX_512bit); + } +} + +static void sub_add(int subResult[], int addResult[], int input1[], int input2[], + MacroAssembler *_masm) { + for (int i = 0; i < 4; i++) { + __ evpsubw(xmm(subResult[i]), k0, xmm(input1[i]), xmm(input2[i]), + false, Assembler::AVX_512bit); + __ evpaddw(xmm(addResult[i]), k0, xmm(input1[i]), xmm(input2[i]), + false, Assembler::AVX_512bit); + } +} + +// result2 also acts as input1 +// result1 also acts as perm1 +static void permute(int result1[], int result2[], int input2[], int perm2, + MacroAssembler *_masm) { + + for (int i = 1; i < 4; i++) { + __ evmovdquw(xmm(result1[i]), xmm(result1[0]), Assembler::AVX_512bit); + } + + for (int i = 0; i < 4; i++) { + __ evpermi2w(xmm(result1[i]), xmm(result2[i]), xmm(input2[i]), + Assembler::AVX_512bit); + __ evpermt2w(xmm(result2[i]), xmm(perm2), xmm(input2[i]), + Assembler::AVX_512bit); + } +} + +static void store4regs(Register address, int offset, int sourceRegs[], + MacroAssembler *_masm) { + for (int i = 0; i < 4; i++) { + __ evmovdquw(Address(address, offset + i * 64), xmm(sourceRegs[i]), + Assembler::AVX_512bit); + } +} + +// In all 3 invocations of this function we use the same registers: +// xmm0-xmm7 for the input and the result, +// xmm8-xmm15 as scratch registers and +// xmm16-xmm17 for the constants, +// so we don't pass register arguments. +static void barrettReduce(MacroAssembler *_masm) { + for (int i = 0; i < 8; i++) { + __ evpmulhw(xmm(i + 8), k0, xmm(i), xmm16, false, Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpsraw(xmm(i + 8), k0, xmm(i + 8), 10, false, Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpmullw(xmm(i + 8), k0, xmm(i + 8), xmm17, false, Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpsubw(xmm(i), k0, xmm(i), xmm(i + 8), false, Assembler::AVX_512bit); + } +} + +static int xmm0_3[] = {0, 1, 2, 3}; +static int xmm0145[] = {0, 1, 4, 5}; +static int xmm0246[] = {0, 2, 4, 6}; +static int xmm0829[] = {0, 8, 2, 9}; +static int xmm1001[] = {1, 0, 0, 1}; +static int xmm1357[] = {1, 3, 5, 7}; +static int xmm2367[] = {2, 3, 6, 7}; +static int xmm2_0_10_8[] = {2, 0, 10, 8}; +static int xmm3223[] = {3, 2, 2, 3}; +static int xmm4_7[] = {4, 5, 6, 7}; +static int xmm5454[] = {5, 4, 5, 4}; +static int xmm7676[] = {7, 6, 7, 6}; +static int xmm8_11[] = {8, 9, 10, 11}; +static int xmm12_15[] = {12, 13, 14, 15}; +static int xmm16_19[] = {16, 17, 18, 19}; +static int xmm20_23[] = {20, 21, 22, 23}; +static int xmm23_23[] = {23, 23, 23, 23}; +static int xmm24_27[] = {24, 25, 26, 27}; +static int xmm26_29[] = {26, 27, 28, 29}; +static int xmm28_31[] = {28, 29, 30, 31}; +static int xmm29_29[] = {29, 29, 29, 29}; + +// Kyber NTT function. +// +// coeffs (short[256]) = c_rarg0 +// ntt_zetas (short[256]) = c_rarg1 +address generate_kyberNtt_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyberNtt_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register coeffs = c_rarg0; + const Register zetas = c_rarg1; + + const Register perms = r11; + + __ lea(perms, ExternalAddress(kyberAvx512NttPermsAddr())); + + load4regs(xmm4_7, coeffs, 256, _masm); + load4regs(xmm20_23, zetas, 0, _masm); + + __ vpbroadcastq(xmm30, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + __ vpbroadcastq(xmm31, + ExternalAddress(kyberAvx512ConstsAddr(qInvModROffset)), + Assembler::AVX_512bit, scratch); // q^-1 mod montR + + load4regs(xmm0_3, coeffs, 0, _masm); + + // Each level represents one iteration of the outer for loop of the Java version. + // level 0 + montmul(xmm8_11, xmm4_7, xmm20_23, xmm8_11, xmm4_7, _masm); + load4regs(xmm20_23, zetas, 256, _masm); + sub_add(xmm4_7, xmm0_3, xmm0_3, xmm8_11, _masm); + + //level 1 + montmul(xmm12_15, xmm2367, xmm20_23, xmm12_15, xmm8_11, _masm); + load4regs(xmm20_23, zetas, 512, _masm); + sub_add(xmm2367, xmm0145, xmm0145, xmm12_15, _masm); + + // level 2 + montmul(xmm8_11, xmm1357, xmm20_23, xmm12_15, xmm8_11, _masm); + __ evmovdquw(xmm12, Address(perms, 0), Assembler::AVX_512bit); + __ evmovdquw(xmm16, Address(perms, 64), Assembler::AVX_512bit); + load4regs(xmm20_23, zetas, 768, _masm); + sub_add(xmm1357, xmm0246, xmm0246, xmm8_11, _masm); + + //level 3 + permute(xmm12_15, xmm0246, xmm1357, 16, _masm); + montmul(xmm8_11, xmm12_15, xmm20_23, xmm16_19, xmm8_11, _masm); + __ evmovdquw(xmm16, Address(perms, 128), Assembler::AVX_512bit); + __ evmovdquw(xmm24, Address(perms, 192), Assembler::AVX_512bit); + load4regs(xmm20_23, zetas, 1024, _masm); + sub_add(xmm1357, xmm0246, xmm0246, xmm8_11, _masm); + + // level 4 + permute(xmm16_19, xmm0246, xmm1357, 24, _masm); + montmul(xmm8_11, xmm0246, xmm20_23, xmm24_27, xmm8_11, _masm); + __ evmovdquw(xmm1, Address(perms, 256), Assembler::AVX_512bit); + __ evmovdquw(xmm24, Address(perms, 320), Assembler::AVX_512bit); + load4regs(xmm20_23, zetas, 1280, _masm); + sub_add(xmm12_15, xmm0246, xmm16_19, xmm8_11, _masm); + + // level 5 + permute(xmm1357, xmm0246, xmm12_15, 24, _masm); + montmul(xmm16_19, xmm0246, xmm20_23, xmm16_19, xmm8_11, _masm); + + __ evmovdquw(xmm12, Address(perms, 384), Assembler::AVX_512bit); + __ evmovdquw(xmm8, Address(perms, 448), Assembler::AVX_512bit); + + load4regs(xmm20_23, zetas, 1536, _masm); + sub_add(xmm24_27, xmm0246, xmm1357, xmm16_19, _masm); + + // level 6 + permute(xmm12_15, xmm0246, xmm24_27, 8, _masm); + + __ evmovdquw(xmm1, Address(perms, 512), Assembler::AVX_512bit); + __ evmovdquw(xmm24, Address(perms, 576), Assembler::AVX_512bit); + + montmul(xmm16_19, xmm0246, xmm20_23, xmm16_19, xmm8_11, _masm); + sub_add(xmm20_23, xmm0246, xmm12_15, xmm16_19, _masm); + + permute(xmm1357, xmm0246, xmm20_23, 24, _masm); + + store4regs(coeffs, 0, xmm0_3, _masm); + store4regs(coeffs, 256, xmm4_7, _masm); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + +// Kyber Inverse NTT function +// +// coeffs (short[256]) = c_rarg0 +// ntt_zetas (short[256]) = c_rarg1 +address generate_kyberInverseNtt_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyberInverseNtt_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register coeffs = c_rarg0; + const Register zetas = c_rarg1; + + const Register perms = r11; + + __ lea(perms, ExternalAddress(kyberAvx512InverseNttPermsAddr())); + __ evmovdquw(xmm12, Address(perms, 0), Assembler::AVX_512bit); + __ evmovdquw(xmm16, Address(perms, 64), Assembler::AVX_512bit); + + __ vpbroadcastq(xmm31, + ExternalAddress(kyberAvx512ConstsAddr(qInvModROffset)), + Assembler::AVX_512bit, scratch); // q^-1 mod montR + __ vpbroadcastq(xmm30, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + __ vpbroadcastq(xmm29, + ExternalAddress(kyberAvx512ConstsAddr(dimHalfInverseOffset)), + Assembler::AVX_512bit, scratch); // (dim/2)^-1 mod q + + load4regs(xmm0_3, coeffs, 0, _masm); + load4regs(xmm4_7, coeffs, 256, _masm); + + // Each level represents one iteration of the outer for loop of the Java version. + // level 0 + load4regs(xmm8_11, zetas, 0, _masm); + permute(xmm12_15, xmm0246, xmm1357, 16, _masm); + + __ evmovdquw(xmm1, Address(perms, 128), Assembler::AVX_512bit); + __ evmovdquw(xmm20, Address(perms, 192), Assembler::AVX_512bit); + + sub_add(xmm16_19, xmm0246, xmm0246, xmm12_15, _masm); + montmul(xmm12_15, xmm16_19, xmm8_11, xmm12_15, xmm8_11, _masm); + + // level 1 + load4regs(xmm8_11, zetas, 256, _masm); + permute(xmm1357, xmm0246, xmm12_15, 20, _masm); + sub_add(xmm16_19, xmm0246, xmm1357, xmm0246, _masm); + + __ evmovdquw(xmm1, Address(perms, 256), Assembler::AVX_512bit); + __ evmovdquw(xmm20, Address(perms, 320), Assembler::AVX_512bit); + + montmul(xmm12_15, xmm16_19, xmm8_11, xmm12_15, xmm8_11, _masm); + + // level2 + load4regs(xmm8_11, zetas, 512, _masm); + permute(xmm1357, xmm0246, xmm12_15, 20, _masm); + sub_add(xmm16_19, xmm0246, xmm1357, xmm0246,_masm); + + __ evmovdquw(xmm1, Address(perms, 384), Assembler::AVX_512bit); + __ evmovdquw(xmm20, Address(perms, 448), Assembler::AVX_512bit); + + montmul(xmm12_15, xmm16_19, xmm8_11, xmm12_15, xmm8_11, _masm); + + __ vpbroadcastq(xmm16, + ExternalAddress(kyberAvx512ConstsAddr(barretMultiplierOffset)), + Assembler::AVX_512bit, scratch); // Barrett multiplier + __ vpbroadcastq(xmm17, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + + permute(xmm1357, xmm0246, xmm12_15, 20, _masm); + barrettReduce(_masm); + +// level 3 + load4regs(xmm8_11, zetas, 768, _masm); + sub_add(xmm16_19, xmm0246, xmm1357, xmm0246, _masm); + + __ evmovdquw(xmm1, Address(perms, 512), Assembler::AVX_512bit); + __ evmovdquw(xmm20, Address(perms, 576), Assembler::AVX_512bit); + + montmul(xmm12_15, xmm16_19, xmm8_11, xmm12_15, xmm8_11, _masm); + permute(xmm1357, xmm0246, xmm12_15, 20, _masm); + + // level 4 + load4regs(xmm8_11, zetas, 1024, _masm); + + __ vpbroadcastq(xmm16, + ExternalAddress(kyberAvx512ConstsAddr(barretMultiplierOffset)), + Assembler::AVX_512bit, scratch); // Barrett multiplier + __ vpbroadcastq(xmm17, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + + sub_add(xmm12_15, xmm0246, xmm0246, xmm1357, _masm); + montmul(xmm1357, xmm12_15, xmm8_11, xmm1357, xmm8_11, _masm); + barrettReduce(_masm); + + // level 5 + load4regs(xmm8_11, zetas, 1280, _masm); + sub_add(xmm12_15, xmm0145, xmm0145, xmm2367, _masm); + montmul(xmm2367, xmm12_15, xmm8_11, xmm2367, xmm8_11, _masm); + + // level 6 + load4regs(xmm8_11, zetas, 1536, _masm); + sub_add(xmm12_15, xmm0_3, xmm0_3, xmm4_7, _masm); + montmul(xmm4_7, xmm12_15, xmm8_11, xmm4_7, xmm8_11, _masm); + + montmul(xmm8_11, xmm29_29, xmm0_3, xmm8_11, xmm0_3, _masm); + montmul(xmm12_15, xmm29_29, xmm4_7, xmm12_15, xmm4_7, _masm); + + store4regs(coeffs, 0, xmm8_11, _masm); + store4regs(coeffs, 256, xmm12_15, _masm); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + +// Kyber multiply polynomials in the NTT domain. +// +// result (short[256]) = c_rarg0 +// ntta (short[256]) = c_rarg1 +// nttb (short[256]) = c_rarg2 +// zetas (short[128]) = c_rarg3 +address generate_kyberNttMult_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyberNttMult_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register result = c_rarg0; + const Register ntta = c_rarg1; + const Register nttb = c_rarg2; + const Register zetas = c_rarg3; + + const Register perms = r11; + const Register loopCnt = r12; + + __ push(r12); + __ movl(loopCnt, 2); + + Label Loop; + + __ lea(perms, ExternalAddress(kyberAvx512_nttMultPermsAddr())); + + + load4regs(xmm26_29, perms, 0, _masm); + __ vpbroadcastq(xmm31, + ExternalAddress(kyberAvx512ConstsAddr(qInvModROffset)), + Assembler::AVX_512bit, scratch); // q^-1 mod montR + __ vpbroadcastq(xmm30, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + __ vpbroadcastq(xmm23, + ExternalAddress(kyberAvx512ConstsAddr(montRSquareModqOffset)), + Assembler::AVX_512bit, scratch); // montR^2 mod q + + __ BIND(Loop); + + __ evmovdquw(xmm1, Address(ntta, 0), Assembler::AVX_512bit); + __ evmovdquw(xmm8, Address(ntta, 64), Assembler::AVX_512bit); + __ evmovdquw(xmm3, Address(ntta, 128), Assembler::AVX_512bit); + __ evmovdquw(xmm9, Address(ntta, 192), Assembler::AVX_512bit); + + __ evmovdquw(xmm5, Address(nttb, 0), Assembler::AVX_512bit); + __ evmovdquw(xmm10, Address(nttb, 64), Assembler::AVX_512bit); + __ evmovdquw(xmm7, Address(nttb, 128), Assembler::AVX_512bit); + __ evmovdquw(xmm11, Address(nttb, 192), Assembler::AVX_512bit); + + __ evmovdquw(xmm0, xmm26, Assembler::AVX_512bit); + __ evmovdquw(xmm2, xmm26, Assembler::AVX_512bit); + __ evmovdquw(xmm4, xmm26, Assembler::AVX_512bit); + __ evmovdquw(xmm6, xmm26, Assembler::AVX_512bit); + + __ evpermi2w(xmm0, xmm1, xmm8, Assembler::AVX_512bit); + __ evpermt2w(xmm1, xmm27, xmm8, Assembler::AVX_512bit); + __ evpermi2w(xmm2, xmm3, xmm9, Assembler::AVX_512bit); + __ evpermt2w(xmm3, xmm27, xmm9, Assembler::AVX_512bit); + + __ evpermi2w(xmm4, xmm5, xmm10, Assembler::AVX_512bit); + __ evpermt2w(xmm5, xmm27, xmm10, Assembler::AVX_512bit); + __ evpermi2w(xmm6, xmm7, xmm11, Assembler::AVX_512bit); + __ evpermt2w(xmm7, xmm27, xmm11, Assembler::AVX_512bit); + + __ evmovdquw(xmm24, Address(zetas, 0), Assembler::AVX_512bit); + __ evmovdquw(xmm25, Address(zetas, 64), Assembler::AVX_512bit); + + montmul(xmm16_19, xmm1001, xmm5454, xmm16_19, xmm12_15, _masm); + + montmul(xmm0145, xmm3223, xmm7676, xmm0145, xmm12_15, _masm); + + __ evpmullw(xmm2, k0, xmm16, xmm24, false, Assembler::AVX_512bit); + __ evpmullw(xmm3, k0, xmm0, xmm25, false, Assembler::AVX_512bit); + __ evpmulhw(xmm12, k0, xmm16, xmm24, false, Assembler::AVX_512bit); + __ evpmulhw(xmm13, k0, xmm0, xmm25, false, Assembler::AVX_512bit); + + __ evpmullw(xmm2, k0, xmm2, xmm31, false, Assembler::AVX_512bit); + __ evpmullw(xmm3, k0, xmm3, xmm31, false, Assembler::AVX_512bit); + __ evpmulhw(xmm2, k0, xmm30, xmm2, false, Assembler::AVX_512bit); + __ evpmulhw(xmm3, k0, xmm30, xmm3, false, Assembler::AVX_512bit); + + __ evpsubw(xmm2, k0, xmm12, xmm2, false, Assembler::AVX_512bit); + __ evpsubw(xmm3, k0, xmm13, xmm3, false, Assembler::AVX_512bit); + + __ evpaddw(xmm0, k0, xmm2, xmm17, false, Assembler::AVX_512bit); + __ evpaddw(xmm8, k0, xmm3, xmm1, false, Assembler::AVX_512bit); + __ evpaddw(xmm2, k0, xmm18, xmm19, false, Assembler::AVX_512bit); + __ evpaddw(xmm9, k0, xmm4, xmm5, false, Assembler::AVX_512bit); + + montmul(xmm1357, xmm0829, xmm23_23, xmm1357, xmm0829, _masm); + + __ evmovdquw(xmm0, xmm28, Assembler::AVX_512bit); + __ evmovdquw(xmm2, xmm28, Assembler::AVX_512bit); + __ evpermi2w(xmm0, xmm1, xmm5, Assembler::AVX_512bit); + __ evpermt2w(xmm1, xmm29, xmm5, Assembler::AVX_512bit); + __ evpermi2w(xmm2, xmm3, xmm7, Assembler::AVX_512bit); + __ evpermt2w(xmm3, xmm29, xmm7, Assembler::AVX_512bit); + + store4regs(result, 0, xmm0_3, _masm); + + __ addptr(ntta, 256); + __ addptr(nttb, 256); + __ addptr(result, 256); + __ addptr(zetas, 128); + __ subl(loopCnt, 1); + __ jcc(Assembler::greater, Loop); + + __ pop(r12); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + +// Kyber add 2 polynomials. +// +// result (short[256]) = c_rarg0 +// a (short[256]) = c_rarg1 +// b (short[256]) = c_rarg2 +address generate_kyberAddPoly_2_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyberAddPoly_2_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register result = c_rarg0; + const Register a = c_rarg1; + const Register b = c_rarg2; + + __ vpbroadcastq(xmm31, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + + for (int i = 0; i < 8; i++) { + __ evmovdquw(xmm(i), Address(a, 64 * i), Assembler::AVX_512bit); + __ evmovdquw(xmm(i + 8), Address(b, 64 * i), Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpaddw(xmm(i), k0, xmm(i), xmm(i + 8), false, Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpaddw(xmm(i), k0, xmm(i), xmm31, false, Assembler::AVX_512bit); + } + + store4regs(result, 0, xmm0_3, _masm); + store4regs(result, 256, xmm4_7, _masm); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + +// Kyber add 3 polynomials. +// +// result (short[256]) = c_rarg0 +// a (short[256]) = c_rarg1 +// b (short[256]) = c_rarg2 +// c (short[256]) = c_rarg3 +address generate_kyberAddPoly_3_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyberAddPoly_3_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register result = c_rarg0; + const Register a = c_rarg1; + const Register b = c_rarg2; + const Register c = c_rarg3; + + __ vpbroadcastq(xmm31, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + + for (int i = 0; i < 8; i++) { + __ evmovdquw(xmm(i), Address(a, 64 * i), Assembler::AVX_512bit); + __ evmovdquw(xmm(i + 8), Address(b, 64 * i), Assembler::AVX_512bit); + __ evmovdquw(xmm(i + 16), Address(c, 64 * i), Assembler::AVX_512bit); + } + + __ evpaddw(xmm31, k0, xmm31, xmm31, false, Assembler::AVX_512bit); + + for (int i = 0; i < 8; i++) { + __ evpaddw(xmm(i), k0, xmm(i), xmm(i + 8), false, Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpaddw(xmm(i), k0, xmm(i), xmm(i + 16), false, Assembler::AVX_512bit); + } + + for (int i = 0; i < 8; i++) { + __ evpaddw(xmm(i), k0, xmm(i), xmm31, false, Assembler::AVX_512bit); + } + + store4regs(result, 0, xmm0_3, _masm); + store4regs(result, 256, xmm4_7, _masm); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + +// Kyber parse XOF output to polynomial coefficient candidates. +// +// condensed (byte[168]) = c_rarg0 +// condensedOffs (int) = c_rarg1 +// parsed (short[112]) = c_rarg2 +// parsedLength (int) = c_rarg3 +address generate_kyber12To16_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyber12To16_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register condensed = c_rarg0; + const Register condensedOffs = c_rarg1; + const Register parsed = c_rarg2; + const Register parsedLength = c_rarg3; + + const Register perms = r11; + + Label Loop; + + __ addptr(condensed, condensedOffs); + + __ lea(perms, ExternalAddress(kyberAvx512_12To16PermsAddr())); + + load4regs(xmm24_27, perms, 0, _masm); + load4regs(xmm28_31, perms, 256, _masm); + __ vpbroadcastq(xmm23, + ExternalAddress(kyberAvx512ConstsAddr(f00Offset)), + Assembler::AVX_512bit, scratch); // 0xF00 + + __ BIND(Loop); + __ evmovdqub(xmm0, Address(condensed, 0),Assembler::AVX_256bit); + __ evmovdqub(xmm1, Address(condensed, 32),Assembler::AVX_256bit); + __ evmovdqub(xmm2, Address(condensed, 64),Assembler::AVX_256bit); + __ evmovdqub(xmm8, Address(condensed, 96),Assembler::AVX_256bit); + __ evmovdqub(xmm9, Address(condensed, 128),Assembler::AVX_256bit); + __ evmovdqub(xmm10, Address(condensed, 160),Assembler::AVX_256bit); + __ vpmovzxbw(xmm0, xmm0, Assembler::AVX_512bit); + __ vpmovzxbw(xmm1, xmm1, Assembler::AVX_512bit); + __ vpmovzxbw(xmm2, xmm2, Assembler::AVX_512bit); + __ vpmovzxbw(xmm8, xmm8, Assembler::AVX_512bit); + __ vpmovzxbw(xmm9, xmm9, Assembler::AVX_512bit); + __ vpmovzxbw(xmm10, xmm10, Assembler::AVX_512bit); + __ evmovdquw(xmm3, xmm24, Assembler::AVX_512bit); + __ evmovdquw(xmm4, xmm25, Assembler::AVX_512bit); + __ evmovdquw(xmm5, xmm26, Assembler::AVX_512bit); + __ evmovdquw(xmm11, xmm24, Assembler::AVX_512bit); + __ evmovdquw(xmm12, xmm25, Assembler::AVX_512bit); + __ evmovdquw(xmm13, xmm26, Assembler::AVX_512bit); + __ evpermi2w(xmm3, xmm0, xmm1, Assembler::AVX_512bit); + __ evpermi2w(xmm4, xmm0, xmm1, Assembler::AVX_512bit); + __ evpermi2w(xmm5, xmm0, xmm1, Assembler::AVX_512bit); + __ evpermi2w(xmm11, xmm8, xmm9, Assembler::AVX_512bit); + __ evpermi2w(xmm12, xmm8, xmm9, Assembler::AVX_512bit); + __ evpermi2w(xmm13, xmm8, xmm9, Assembler::AVX_512bit); + __ evpermt2w(xmm3, xmm27, xmm2, Assembler::AVX_512bit); + __ evpermt2w(xmm4, xmm28, xmm2, Assembler::AVX_512bit); + __ evpermt2w(xmm5, xmm29, xmm2, Assembler::AVX_512bit); + __ evpermt2w(xmm11, xmm27, xmm10, Assembler::AVX_512bit); + __ evpermt2w(xmm12, xmm28, xmm10, Assembler::AVX_512bit); + __ evpermt2w(xmm13, xmm29, xmm10, Assembler::AVX_512bit); + + __ evpsraw(xmm2, k0, xmm4, 4, false, Assembler::AVX_512bit); + __ evpsllw(xmm0, k0, xmm4, 8, false, Assembler::AVX_512bit); + __ evpsllw(xmm1, k0, xmm5, 4, false, Assembler::AVX_512bit); + __ evpsllw(xmm8, k0, xmm12, 8, false, Assembler::AVX_512bit); + __ evpsraw(xmm10, k0, xmm12, 4, false, Assembler::AVX_512bit); + __ evpsllw(xmm9, k0, xmm13, 4, false, Assembler::AVX_512bit); + __ evpandq(xmm0, k0, xmm0, xmm23, false, Assembler::AVX_512bit); + __ evpandq(xmm8, k0, xmm8, xmm23, false, Assembler::AVX_512bit); + __ evpaddw(xmm1, k0, xmm1, xmm2, false, Assembler::AVX_512bit); + __ evpaddw(xmm0, k0, xmm0, xmm3, false, Assembler::AVX_512bit); + __ evmovdquw(xmm2, xmm30, Assembler::AVX_512bit); + __ evpaddw(xmm9, k0, xmm9, xmm10, false, Assembler::AVX_512bit); + __ evpaddw(xmm8, k0, xmm8, xmm11, false, Assembler::AVX_512bit); + __ evmovdquw(xmm10, xmm30, Assembler::AVX_512bit); + __ evpermi2w(xmm2, xmm0, xmm1, Assembler::AVX_512bit); + __ evpermt2w(xmm0, xmm31, xmm1, Assembler::AVX_512bit); + __ evpermi2w(xmm10, xmm8, xmm9, Assembler::AVX_512bit); + __ evpermt2w(xmm8, xmm31, xmm9, Assembler::AVX_512bit); + + store4regs(parsed, 0, xmm2_0_10_8, _masm); + + __ addptr(condensed, 192); + __ addptr(parsed, 256); + __ subl(parsedLength, 128); + __ jcc(Assembler::greater, Loop); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + + +// Kyber barrett reduce function. +// +// coeffs (short[256]) = c_rarg0 +address generate_kyberBarrettReduce_avx512(StubGenerator *stubgen, + MacroAssembler *_masm) { + + __ align(CodeEntryAlignment); + StubGenStubId stub_id = kyberBarrettReduce_id; + StubCodeMark mark(stubgen, stub_id); + address start = __ pc(); + __ enter(); + + const Register coeffs = c_rarg0; + + __ vpbroadcastq(xmm16, + ExternalAddress(kyberAvx512ConstsAddr(barretMultiplierOffset)), + Assembler::AVX_512bit, scratch); // Barrett multiplier + __ vpbroadcastq(xmm17, + ExternalAddress(kyberAvx512ConstsAddr(qOffset)), + Assembler::AVX_512bit, scratch); // q + + load4regs(xmm0_3, coeffs, 0, _masm); + load4regs(xmm4_7, coeffs, 256, _masm); + + barrettReduce(_masm); + + store4regs(coeffs, 0, xmm0_3, _masm); + store4regs(coeffs, 256, xmm4_7, _masm); + + __ leave(); // required for proper stackwalking of RuntimeStub frame + __ mov64(rax, 0); // return 0 + __ ret(0); + + return start; +} + +void StubGenerator::generate_kyber_stubs() { + // Generate Kyber intrinsics code + if (UseKyberIntrinsics) { + if (VM_Version::supports_evex()) { + StubRoutines::_kyberNtt = generate_kyberNtt_avx512(this, _masm); + StubRoutines::_kyberInverseNtt = generate_kyberInverseNtt_avx512(this, _masm); + StubRoutines::_kyberNttMult = generate_kyberNttMult_avx512(this, _masm); + StubRoutines::_kyberAddPoly_2 = generate_kyberAddPoly_2_avx512(this, _masm); + StubRoutines::_kyberAddPoly_3 = generate_kyberAddPoly_3_avx512(this, _masm); + StubRoutines::_kyber12To16 = generate_kyber12To16_avx512(this, _masm); + StubRoutines::_kyberBarrettReduce = generate_kyberBarrettReduce_avx512(this, _masm); + } + } +} diff --git a/src/hotspot/cpu/x86/stubGenerator_x86_64_tanh.cpp b/src/hotspot/cpu/x86/stubGenerator_x86_64_tanh.cpp index d13809bfcd9..52ce2731b1f 100644 --- a/src/hotspot/cpu/x86/stubGenerator_x86_64_tanh.cpp +++ b/src/hotspot/cpu/x86/stubGenerator_x86_64_tanh.cpp @@ -1,5 +1,5 @@ /* -* Copyright (c) 2024, Intel Corporation. All rights reserved. +* Copyright (c) 2024, 2025, Intel Corporation. All rights reserved. * Intel Math Library (LIBM) Source Code * * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. @@ -46,7 +46,7 @@ // for |x| in [23/64,3*2^7) // e^{-2*|x|}=2^{-k-f}*2^{-r} ~ 2^{-k}*(Tn+Dn)*(1+p)=(T0+D0)*(1+p) // -// For |x| in [2^{-4},2^5): +// For |x| in [2^{-4},22): // 2^{-r}-1 ~ p=c1*r+c2*r^2+..+c5*r^5 // Let R=1/(1+T0+p*T0), truncated to 35 significant bits // R=1/(1+T0+D0+p*(T0+D0))*(1+eps), |eps|<2^{-33} @@ -66,11 +66,11 @@ // // For |x|<2^{-64}: x is returned // -// For |x|>=2^32: return +/-1 +// For |x|>=22: return +/-1 // // Special cases: // tanh(NaN) = quiet NaN, and raise invalid exception -// tanh(INF) = that INF +// tanh(+/-INF) = +/-1 // tanh(+/-0) = +/-0 // /******************************************************************************/ @@ -324,6 +324,12 @@ address StubGenerator::generate_libmTanh() { __ enter(); // required for proper stackwalking of RuntimeStub frame __ bind(B1_2); + __ pextrw(rcx, xmm0, 3); + __ movl(rdx, 32768); + __ andl(rdx, rcx); + __ andl(rcx, 32767); + __ cmpl(rcx, 16438); + __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_2_0_1); // Branch only if |x| >= 22 __ movsd(xmm3, ExternalAddress(HALFMASK), r11 /*rscratch*/); __ xorpd(xmm4, xmm4); __ movsd(xmm1, ExternalAddress(L2E), r11 /*rscratch*/); @@ -331,16 +337,12 @@ address StubGenerator::generate_libmTanh() { __ movl(rax, 32768); __ pinsrw(xmm4, rax, 3); __ movsd(xmm6, ExternalAddress(Shifter), r11 /*rscratch*/); - __ pextrw(rcx, xmm0, 3); __ andpd(xmm3, xmm0); __ andnpd(xmm4, xmm0); __ pshufd(xmm5, xmm4, 68); - __ movl(rdx, 32768); - __ andl(rdx, rcx); - __ andl(rcx, 32767); __ subl(rcx, 16304); - __ cmpl(rcx, 144); - __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_0_0_1); + __ cmpl(rcx, 134); + __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_0_0_1); // Branch only if |x| is not in [2^{-4},22) __ subsd(xmm4, xmm3); __ mulsd(xmm3, xmm1); __ mulsd(xmm2, xmm5); @@ -427,8 +429,8 @@ address StubGenerator::generate_libmTanh() { __ bind(L_2TAG_PACKET_0_0_1); __ addl(rcx, 960); - __ cmpl(rcx, 1104); - __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_1_0_1); + __ cmpl(rcx, 1094); + __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_1_0_1); // Branch only if |x| not in [2^{-64}, 2^{-4}) __ movdqu(xmm2, ExternalAddress(pv), r11 /*rscratch*/); __ pshufd(xmm1, xmm0, 68); __ movdqu(xmm3, ExternalAddress(pv + 16), r11 /*rscratch*/); @@ -449,11 +451,8 @@ address StubGenerator::generate_libmTanh() { __ jmp(B1_4); __ bind(L_2TAG_PACKET_1_0_1); - __ addl(rcx, 15344); - __ cmpl(rcx, 16448); - __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_2_0_1); __ cmpl(rcx, 16); - __ jcc(Assembler::below, L_2TAG_PACKET_3_0_1); + __ jcc(Assembler::below, L_2TAG_PACKET_3_0_1); // Branch only if |x| is denormalized __ xorpd(xmm2, xmm2); __ movl(rax, 17392); __ pinsrw(xmm2, rax, 3); @@ -468,7 +467,7 @@ address StubGenerator::generate_libmTanh() { __ bind(L_2TAG_PACKET_2_0_1); __ cmpl(rcx, 32752); - __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_4_0_1); + __ jcc(Assembler::aboveEqual, L_2TAG_PACKET_4_0_1); // Branch only if |x| is INF or NaN __ xorpd(xmm2, xmm2); __ movl(rcx, 15344); __ pinsrw(xmm2, rcx, 3); @@ -489,7 +488,7 @@ address StubGenerator::generate_libmTanh() { __ movdl(rcx, xmm2); __ orl(rcx, rax); __ cmpl(rcx, 0); - __ jcc(Assembler::equal, L_2TAG_PACKET_5_0_1); + __ jcc(Assembler::equal, L_2TAG_PACKET_5_0_1); // Branch only if |x| is not NaN __ addsd(xmm0, xmm0); __ bind(B1_4); diff --git a/src/hotspot/cpu/x86/templateInterpreterGenerator_x86.cpp b/src/hotspot/cpu/x86/templateInterpreterGenerator_x86.cpp index 45e30a8b4fb..bd061d45fbd 100644 --- a/src/hotspot/cpu/x86/templateInterpreterGenerator_x86.cpp +++ b/src/hotspot/cpu/x86/templateInterpreterGenerator_x86.cpp @@ -1147,6 +1147,30 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) { __ bind(L); } +#if INCLUDE_JFR + __ enter_jfr_critical_section(); + + // This poll test is to uphold the invariant that a JFR sampled frame + // must not return to its caller without a prior safepoint poll check. + // The earlier poll check in this routine is insufficient for this purpose + // because the thread has transitioned back to Java. + + Label slow_path; + Label fast_path; + __ safepoint_poll(slow_path, true /* at_return */, false /* in_nmethod */); + __ jmp(fast_path); + __ bind(slow_path); + __ push(dtos); + __ push(ltos); + __ set_last_Java_frame(noreg, rbp, (address)__ pc(), rscratch1); + __ super_call_VM_leaf(CAST_FROM_FN_PTR(address, InterpreterRuntime::at_unwind), r15_thread); + __ reset_last_Java_frame(true); + __ pop(ltos); + __ pop(dtos); + __ bind(fast_path); + +#endif // INCLUDE_JFR + // jvmti support // Note: This must happen _after_ handling/throwing any exceptions since // the exception handler code notifies the runtime of method exits @@ -1169,8 +1193,12 @@ address TemplateInterpreterGenerator::generate_native_entry(bool synchronized) { frame::interpreter_frame_sender_sp_offset * wordSize)); // get sender sp __ leave(); // remove frame anchor + + JFR_ONLY(__ leave_jfr_critical_section();) + __ pop(rdi); // get return address __ mov(rsp, t); // set sp to sender sp + __ jmp(rdi); if (inc_counter) { @@ -1413,6 +1441,7 @@ void TemplateInterpreterGenerator::generate_throw_exception() { Interpreter::_remove_activation_preserving_args_entry = __ pc(); __ empty_expression_stack(); + __ restore_bcp(); // We could have returned from deoptimizing this frame, so restore rbcp. // Set the popframe_processing bit in pending_popframe_condition // indicating that we are currently handling popframe, so that // call_VMs that may happen later do not trigger new popframe diff --git a/src/hotspot/cpu/x86/templateInterpreterGenerator_x86_64.cpp b/src/hotspot/cpu/x86/templateInterpreterGenerator_x86_64.cpp index 6be702f2699..9ea4aeeccfa 100644 --- a/src/hotspot/cpu/x86/templateInterpreterGenerator_x86_64.cpp +++ b/src/hotspot/cpu/x86/templateInterpreterGenerator_x86_64.cpp @@ -468,6 +468,10 @@ address TemplateInterpreterGenerator::generate_math_entry(AbstractInterpreter::M assert(StubRoutines::dtanh() != nullptr, "not initialized"); __ movdbl(xmm0, Address(rsp, wordSize)); __ call(RuntimeAddress(CAST_FROM_FN_PTR(address, StubRoutines::dtanh()))); + } else if (kind == Interpreter::java_lang_math_cbrt) { + assert(StubRoutines::dcbrt() != nullptr, "not initialized"); + __ movdbl(xmm0, Address(rsp, wordSize)); + __ call(RuntimeAddress(CAST_FROM_FN_PTR(address, StubRoutines::dcbrt()))); } else if (kind == Interpreter::java_lang_math_abs) { assert(StubRoutines::x86::double_sign_mask() != nullptr, "not initialized"); __ movdbl(xmm0, Address(rsp, wordSize)); diff --git a/src/hotspot/cpu/x86/templateTable_x86.cpp b/src/hotspot/cpu/x86/templateTable_x86.cpp index 43da80f4082..82ca18d8a1f 100644 --- a/src/hotspot/cpu/x86/templateTable_x86.cpp +++ b/src/hotspot/cpu/x86/templateTable_x86.cpp @@ -1687,8 +1687,7 @@ void TemplateTable::float_cmp(bool is_float, int unordered_result) { void TemplateTable::branch(bool is_jsr, bool is_wide) { __ get_method(rcx); // rcx holds method - __ profile_taken_branch(rax, rbx); // rax holds updated MDP, rbx - // holds bumped taken count + __ profile_taken_branch(rax); // rax holds updated MDP const ByteSize be_offset = MethodCounters::backedge_counter_offset() + InvocationCounter::counter_offset(); @@ -1739,7 +1738,6 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) { if (UseLoopCounter) { // increment backedge counter for backward branches // rax: MDO - // rbx: MDO bumped taken-count // rcx: method // rdx: target offset // r13: target bcp @@ -1825,6 +1823,8 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) { // it will be preserved in rbx. __ mov(rbx, rax); + JFR_ONLY(__ enter_jfr_critical_section();) + call_VM(noreg, CAST_FROM_FN_PTR(address, SharedRuntime::OSR_migration_begin)); // rax is OSR buffer, move it to expected parameter location @@ -1839,14 +1839,12 @@ void TemplateTable::branch(bool is_jsr, bool is_wide) { // pop the interpreter frame __ movptr(sender_sp, Address(rbp, frame::interpreter_frame_sender_sp_offset * wordSize)); // get sender sp __ leave(); // remove frame anchor + JFR_ONLY(__ leave_jfr_critical_section();) __ pop(retaddr); // get return address - __ mov(rsp, sender_sp); // set sp to sender sp + __ mov(rsp, sender_sp); // set sp to sender sp // Ensure compiled code always sees stack at proper alignment __ andptr(rsp, -(StackAlignmentInBytes)); - // unlike x86 we need no specialized return from compiled code - // to the interpreter or the call stub. - // push the return address __ push(retaddr); diff --git a/src/hotspot/cpu/x86/vmStructs_x86.hpp b/src/hotspot/cpu/x86/vmStructs_x86.hpp index d894d8b09a7..b8089a6413e 100644 --- a/src/hotspot/cpu/x86/vmStructs_x86.hpp +++ b/src/hotspot/cpu/x86/vmStructs_x86.hpp @@ -29,15 +29,20 @@ // constants required by the Serviceability Agent. This file is // referenced by vmStructs.cpp. -#define VM_STRUCTS_CPU(nonstatic_field, static_field, unchecked_nonstatic_field, volatile_nonstatic_field, nonproduct_nonstatic_field) \ - volatile_nonstatic_field(JavaFrameAnchor, _last_Java_fp, intptr_t*) +#define VM_STRUCTS_CPU(nonstatic_field, static_field, unchecked_nonstatic_field, volatile_nonstatic_field, nonproduct_nonstatic_field) \ + volatile_nonstatic_field(JavaFrameAnchor, _last_Java_fp, intptr_t*) \ + static_field(VM_Version, _features, VM_Version::VM_Features) \ + nonstatic_field(VM_Version::VM_Features, _features_bitmap[0], uint64_t) \ + static_field(VM_Version::VM_Features, _features_bitmap_size, int) #define VM_TYPES_CPU(declare_type, declare_toplevel_type, declare_oop_type, declare_integer_type, declare_unsigned_integer_type) \ + declare_toplevel_type(VM_Version::VM_Features) #define VM_INT_CONSTANTS_CPU(declare_constant, declare_preprocessor_constant) \ - LP64_ONLY(declare_constant(frame::arg_reg_save_area_bytes)) \ - declare_constant(frame::interpreter_frame_sender_sp_offset) \ - declare_constant(frame::interpreter_frame_last_sp_offset) + declare_constant(frame::arg_reg_save_area_bytes) \ + declare_constant(frame::interpreter_frame_sender_sp_offset) \ + declare_constant(frame::interpreter_frame_last_sp_offset) \ + declare_constant(frame::entry_frame_call_wrapper_offset) #define VM_LONG_CONSTANTS_CPU(declare_constant, declare_preprocessor_constant) diff --git a/src/hotspot/cpu/x86/vm_version_x86.cpp b/src/hotspot/cpu/x86/vm_version_x86.cpp index 32e6e33d133..152866e65f3 100644 --- a/src/hotspot/cpu/x86/vm_version_x86.cpp +++ b/src/hotspot/cpu/x86/vm_version_x86.cpp @@ -63,6 +63,11 @@ address VM_Version::_cpuinfo_cont_addr_apx = nullptr; static BufferBlob* stub_blob; static const int stub_size = 2000; +int VM_Version::VM_Features::_features_bitmap_size = sizeof(VM_Version::VM_Features::_features_bitmap) / BytesPerLong; + +VM_Version::VM_Features VM_Version::_features; +VM_Version::VM_Features VM_Version::_cpu_features; + extern "C" { typedef void (*get_cpu_info_stub_t)(void*); typedef void (*detect_virt_stub_t)(uint32_t, uint32_t*); @@ -72,8 +77,6 @@ static get_cpu_info_stub_t get_cpu_info_stub = nullptr; static detect_virt_stub_t detect_virt_stub = nullptr; static clear_apx_test_state_t clear_apx_test_state_stub = nullptr; -#ifdef _LP64 - bool VM_Version::supports_clflush() { // clflush should always be available on x86_64 // if not we are in real trouble because we rely on it @@ -84,10 +87,9 @@ bool VM_Version::supports_clflush() { // up. Assembler::flush calls this routine to check that clflush // is allowed. So, we give the caller a free pass if Universe init // is still in progress. - assert ((!Universe::is_fully_initialized() || (_features & CPU_FLUSH) != 0), "clflush should be available"); + assert ((!Universe::is_fully_initialized() || _features.supports_feature(CPU_FLUSH)), "clflush should be available"); return true; } -#endif #define CPUID_STANDARD_FN 0x0 #define CPUID_STANDARD_FN_1 0x1 @@ -107,7 +109,6 @@ class VM_Version_StubGenerator: public StubCodeGenerator { VM_Version_StubGenerator(CodeBuffer *c) : StubCodeGenerator(c) {} -#if defined(_LP64) address clear_apx_test_state() { # define __ _masm-> address start = __ pc(); @@ -126,7 +127,6 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ ret(0); return start; } -#endif address generate_get_cpu_info() { // Flags to test CPU type. @@ -138,7 +138,7 @@ class VM_Version_StubGenerator: public StubCodeGenerator { const uint32_t CPU_FAMILY_486 = (4 << CPU_FAMILY_SHIFT); bool use_evex = FLAG_IS_DEFAULT(UseAVX) || (UseAVX > 2); - Label detect_486, cpu486, detect_586, std_cpuid1, std_cpuid4; + Label detect_486, cpu486, detect_586, std_cpuid1, std_cpuid4, std_cpuid24; Label sef_cpuid, sefsl1_cpuid, ext_cpuid, ext_cpuid1, ext_cpuid5, ext_cpuid7; Label ext_cpuid8, done, wrapup, vector_save_restore, apx_save_restore_warning; Label legacy_setup, save_restore_except, legacy_save_restore, start_simd_check; @@ -151,14 +151,10 @@ class VM_Version_StubGenerator: public StubCodeGenerator { // // void get_cpu_info(VM_Version::CpuidInfo* cpuid_info); // - // LP64: rcx and rdx are first and second argument registers on windows + // rcx and rdx are first and second argument registers on windows __ push(rbp); -#ifdef _LP64 __ mov(rbp, c_rarg0); // cpuid_info address -#else - __ movptr(rbp, Address(rsp, 8)); // cpuid_info address -#endif __ push(rbx); __ push(rsi); __ pushf(); // preserve rbx, and flags @@ -341,6 +337,17 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ movl(Address(rsi, 0), rax); __ movl(Address(rsi, 4), rdx); + // + // cpuid(0x24) Converged Vector ISA Main Leaf (EAX = 24H, ECX = 0). + // + __ bind(std_cpuid24); + __ movl(rax, 0x24); + __ movl(rcx, 0); + __ cpuid(); + __ lea(rsi, Address(rbp, in_bytes(VM_Version::std_cpuid24_offset()))); + __ movl(Address(rsi, 0), rax); + __ movl(Address(rsi, 4), rbx); + // // Extended cpuid(0x80000000) // @@ -418,7 +425,6 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ movl(Address(rsi, 8), rcx); __ movl(Address(rsi,12), rdx); -#if defined(_LP64) // // Check if OS has enabled XGETBV instruction to access XCR0 // (OSXSAVE feature flag) and CPU supports APX @@ -428,13 +434,11 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ lea(rsi, Address(rbp, in_bytes(VM_Version::sefsl1_cpuid7_offset()))); __ movl(rax, 0x200000); __ andl(rax, Address(rsi, 4)); - __ cmpl(rax, 0x200000); - __ jcc(Assembler::notEqual, vector_save_restore); + __ jcc(Assembler::equal, vector_save_restore); // check _cpuid_info.xem_xcr0_eax.bits.apx_f __ movl(rax, 0x80000); __ andl(rax, Address(rbp, in_bytes(VM_Version::xem_xcr0_offset()))); // xcr0 bits apx_f - __ cmpl(rax, 0x80000); - __ jcc(Assembler::notEqual, vector_save_restore); + __ jcc(Assembler::equal, vector_save_restore); #ifndef PRODUCT bool save_apx = UseAPX; @@ -453,7 +457,6 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ movq(Address(rsi, 8), r31); UseAPX = save_apx; -#endif #endif __ bind(vector_save_restore); // @@ -488,11 +491,15 @@ class VM_Version_StubGenerator: public StubCodeGenerator { // If UseAVX is uninitialized or is set by the user to include EVEX if (use_evex) { // check _cpuid_info.sef_cpuid7_ebx.bits.avx512f + // OR check _cpuid_info.sefsl1_cpuid7_edx.bits.avx10 __ lea(rsi, Address(rbp, in_bytes(VM_Version::sef_cpuid7_offset()))); __ movl(rax, 0x10000); - __ andl(rax, Address(rsi, 4)); // xcr0 bits sse | ymm - __ cmpl(rax, 0x10000); - __ jccb(Assembler::notEqual, legacy_setup); // jump if EVEX is not supported + __ andl(rax, Address(rsi, 4)); + __ lea(rsi, Address(rbp, in_bytes(VM_Version::sefsl1_cpuid7_offset()))); + __ movl(rbx, 0x80000); + __ andl(rbx, Address(rsi, 4)); + __ orl(rax, rbx); + __ jccb(Assembler::equal, legacy_setup); // jump if EVEX is not supported // check _cpuid_info.xem_xcr0_eax.bits.opmask // check _cpuid_info.xem_xcr0_eax.bits.zmm512 // check _cpuid_info.xem_xcr0_eax.bits.zmm32 @@ -527,10 +534,8 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ movdl(xmm0, rcx); __ vpbroadcastd(xmm0, xmm0, Assembler::AVX_512bit); __ evmovdqul(xmm7, xmm0, Assembler::AVX_512bit); -#ifdef _LP64 __ evmovdqul(xmm8, xmm0, Assembler::AVX_512bit); __ evmovdqul(xmm31, xmm0, Assembler::AVX_512bit); -#endif VM_Version::clean_cpuFeatures(); __ jmp(save_restore_except); } @@ -556,10 +561,8 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ pshufd(xmm0, xmm0, 0x00); __ vinsertf128_high(xmm0, xmm0); __ vmovdqu(xmm7, xmm0); -#ifdef _LP64 __ vmovdqu(xmm8, xmm0); __ vmovdqu(xmm15, xmm0); -#endif VM_Version::clean_cpuFeatures(); __ bind(save_restore_except); @@ -577,8 +580,7 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ lea(rsi, Address(rbp, in_bytes(VM_Version::sef_cpuid7_offset()))); __ movl(rax, 0x10000); __ andl(rax, Address(rsi, 4)); - __ cmpl(rax, 0x10000); - __ jcc(Assembler::notEqual, legacy_save_restore); + __ jcc(Assembler::equal, legacy_save_restore); // check _cpuid_info.xem_xcr0_eax.bits.opmask // check _cpuid_info.xem_xcr0_eax.bits.zmm512 // check _cpuid_info.xem_xcr0_eax.bits.zmm32 @@ -600,10 +602,8 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ lea(rsi, Address(rbp, in_bytes(VM_Version::zmm_save_offset()))); __ evmovdqul(Address(rsi, 0), xmm0, Assembler::AVX_512bit); __ evmovdqul(Address(rsi, 64), xmm7, Assembler::AVX_512bit); -#ifdef _LP64 __ evmovdqul(Address(rsi, 128), xmm8, Assembler::AVX_512bit); __ evmovdqul(Address(rsi, 192), xmm31, Assembler::AVX_512bit); -#endif #ifdef _WINDOWS __ evmovdqul(xmm31, Address(rsp, 0), Assembler::AVX_512bit); @@ -628,10 +628,8 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ lea(rsi, Address(rbp, in_bytes(VM_Version::ymm_save_offset()))); __ vmovdqu(Address(rsi, 0), xmm0); __ vmovdqu(Address(rsi, 32), xmm7); -#ifdef _LP64 __ vmovdqu(Address(rsi, 64), xmm8); __ vmovdqu(Address(rsi, 96), xmm15); -#endif #ifdef _WINDOWS __ vmovdqu(xmm15, Address(rsp, 0)); @@ -687,13 +685,8 @@ class VM_Version_StubGenerator: public StubCodeGenerator { __ push(rbx); __ push(rsi); // for Windows -#ifdef _LP64 __ mov(rax, c_rarg0); // CPUID leaf __ mov(rsi, c_rarg1); // register array address (eax, ebx, ecx, edx) -#else - __ movptr(rax, Address(rsp, 16)); // CPUID leaf - __ movptr(rsi, Address(rsp, 20)); // register array address -#endif __ cpuid(); @@ -734,14 +727,10 @@ class VM_Version_StubGenerator: public StubCodeGenerator { // // void getCPUIDBrandString(VM_Version::CpuidInfo* cpuid_info); // - // LP64: rcx and rdx are first and second argument registers on windows + // rcx and rdx are first and second argument registers on windows __ push(rbp); -#ifdef _LP64 __ mov(rbp, c_rarg0); // cpuid_info address -#else - __ movptr(rbp, Address(rsp, 8)); // cpuid_info address -#endif __ push(rbx); __ push(rsi); __ pushf(); // preserve rbx, and flags @@ -863,7 +852,6 @@ void VM_Version::get_processor_features() { _cpu = 4; // 486 by default _model = 0; _stepping = 0; - _features = 0; _logical_processors_per_package = 1; // i486 internal cache is both I&D and has a 16-byte line size _L1_data_cache_line_size = 16; @@ -879,7 +867,7 @@ void VM_Version::get_processor_features() { if (cpu_family() > 4) { // it supports CPUID _features = _cpuid_info.feature_flags(); // These can be changed by VM settings - _cpu_features = _features; // Preserve features + _cpu_features = _features; // Preserve features // Logical processors are only available on P4s and above, // and only if hyperthreading is available. _logical_processors_per_package = logical_processor_count(); @@ -889,19 +877,16 @@ void VM_Version::get_processor_features() { // xchg and xadd instructions _supports_atomic_getset4 = true; _supports_atomic_getadd4 = true; - LP64_ONLY(_supports_atomic_getset8 = true); - LP64_ONLY(_supports_atomic_getadd8 = true); + _supports_atomic_getset8 = true; + _supports_atomic_getadd8 = true; -#ifdef _LP64 // OS should support SSE for x64 and hardware should support at least SSE2. if (!VM_Version::supports_sse2()) { vm_exit_during_initialization("Unknown x64 processor: SSE2 not supported"); } // in 64 bit the use of SSE2 is the minimum if (UseSSE < 2) UseSSE = 2; -#endif -#ifdef AMD64 // flush_icache_stub have to be generated first. // That is why Icache line size is hard coded in ICache class, // see icache_x86.hpp. It is also the reason why we can't use @@ -913,9 +898,7 @@ void VM_Version::get_processor_features() { guarantee(_cpuid_info.std_cpuid1_edx.bits.clflush != 0, "clflush is not supported"); // clflush_size is size in quadwords (8 bytes). guarantee(_cpuid_info.std_cpuid1_ebx.bits.clflush_size == 8, "such clflush size is not supported"); -#endif -#ifdef _LP64 // assigning this field effectively enables Unsafe.writebackMemory() // by initing UnsafeConstant.DATA_CACHE_LINE_FLUSH_SIZE to non-zero // that is only implemented on x86_64 and only if the OS plays ball @@ -924,31 +907,30 @@ void VM_Version::get_processor_features() { // let if default to zero thereby disabling writeback _data_cache_line_flush_size = _cpuid_info.std_cpuid1_ebx.bits.clflush_size * 8; } -#endif // Check if processor has Intel Ecore - if (FLAG_IS_DEFAULT(EnableX86ECoreOpts) && is_intel() && cpu_family() == 6 && + if (FLAG_IS_DEFAULT(EnableX86ECoreOpts) && is_intel() && is_intel_server_family() && (_model == 0x97 || _model == 0xAA || _model == 0xAC || _model == 0xAF || _model == 0xCC || _model == 0xDD)) { FLAG_SET_DEFAULT(EnableX86ECoreOpts, true); } if (UseSSE < 4) { - _features &= ~CPU_SSE4_1; - _features &= ~CPU_SSE4_2; + _features.clear_feature(CPU_SSE4_1); + _features.clear_feature(CPU_SSE4_2); } if (UseSSE < 3) { - _features &= ~CPU_SSE3; - _features &= ~CPU_SSSE3; - _features &= ~CPU_SSE4A; + _features.clear_feature(CPU_SSE3); + _features.clear_feature(CPU_SSSE3); + _features.clear_feature(CPU_SSE4A); } if (UseSSE < 2) - _features &= ~CPU_SSE2; + _features.clear_feature(CPU_SSE2); if (UseSSE < 1) - _features &= ~CPU_SSE; + _features.clear_feature(CPU_SSE); //since AVX instructions is slower than SSE in some ZX cpus, force USEAVX=0. if (is_zx() && ((cpu_family() == 6) || (cpu_family() == 7))) { @@ -1014,21 +996,25 @@ void VM_Version::get_processor_features() { } if (UseAVX < 3) { - _features &= ~CPU_AVX512F; - _features &= ~CPU_AVX512DQ; - _features &= ~CPU_AVX512CD; - _features &= ~CPU_AVX512BW; - _features &= ~CPU_AVX512VL; - _features &= ~CPU_AVX512_VPOPCNTDQ; - _features &= ~CPU_AVX512_VPCLMULQDQ; - _features &= ~CPU_AVX512_VAES; - _features &= ~CPU_AVX512_VNNI; - _features &= ~CPU_AVX512_VBMI; - _features &= ~CPU_AVX512_VBMI2; - _features &= ~CPU_AVX512_BITALG; - _features &= ~CPU_AVX512_IFMA; - _features &= ~CPU_APX_F; - _features &= ~CPU_AVX512_FP16; + _features.clear_feature(CPU_AVX512F); + _features.clear_feature(CPU_AVX512DQ); + _features.clear_feature(CPU_AVX512CD); + _features.clear_feature(CPU_AVX512BW); + _features.clear_feature(CPU_AVX512ER); + _features.clear_feature(CPU_AVX512PF); + _features.clear_feature(CPU_AVX512VL); + _features.clear_feature(CPU_AVX512_VPOPCNTDQ); + _features.clear_feature(CPU_AVX512_VPCLMULQDQ); + _features.clear_feature(CPU_AVX512_VAES); + _features.clear_feature(CPU_AVX512_VNNI); + _features.clear_feature(CPU_AVX512_VBMI); + _features.clear_feature(CPU_AVX512_VBMI2); + _features.clear_feature(CPU_AVX512_BITALG); + _features.clear_feature(CPU_AVX512_IFMA); + _features.clear_feature(CPU_APX_F); + _features.clear_feature(CPU_AVX512_FP16); + _features.clear_feature(CPU_AVX10_1); + _features.clear_feature(CPU_AVX10_2); } // Currently APX support is only enabled for targets supporting AVX512VL feature. @@ -1041,45 +1027,47 @@ void VM_Version::get_processor_features() { } if (!UseAPX) { - _features &= ~CPU_APX_F; + _features.clear_feature(CPU_APX_F); } if (UseAVX < 2) { - _features &= ~CPU_AVX2; - _features &= ~CPU_AVX_IFMA; + _features.clear_feature(CPU_AVX2); + _features.clear_feature(CPU_AVX_IFMA); } if (UseAVX < 1) { - _features &= ~CPU_AVX; - _features &= ~CPU_VZEROUPPER; - _features &= ~CPU_F16C; - _features &= ~CPU_SHA512; + _features.clear_feature(CPU_AVX); + _features.clear_feature(CPU_VZEROUPPER); + _features.clear_feature(CPU_F16C); + _features.clear_feature(CPU_SHA512); } if (logical_processors_per_package() == 1) { // HT processor could be installed on a system which doesn't support HT. - _features &= ~CPU_HT; + _features.clear_feature(CPU_HT); } if (is_intel()) { // Intel cpus specific settings if (is_knights_family()) { - _features &= ~CPU_VZEROUPPER; - _features &= ~CPU_AVX512BW; - _features &= ~CPU_AVX512VL; - _features &= ~CPU_AVX512DQ; - _features &= ~CPU_AVX512_VNNI; - _features &= ~CPU_AVX512_VAES; - _features &= ~CPU_AVX512_VPOPCNTDQ; - _features &= ~CPU_AVX512_VPCLMULQDQ; - _features &= ~CPU_AVX512_VBMI; - _features &= ~CPU_AVX512_VBMI2; - _features &= ~CPU_CLWB; - _features &= ~CPU_FLUSHOPT; - _features &= ~CPU_GFNI; - _features &= ~CPU_AVX512_BITALG; - _features &= ~CPU_AVX512_IFMA; - _features &= ~CPU_AVX_IFMA; - _features &= ~CPU_AVX512_FP16; + _features.clear_feature(CPU_VZEROUPPER); + _features.clear_feature(CPU_AVX512BW); + _features.clear_feature(CPU_AVX512VL); + _features.clear_feature(CPU_AVX512DQ); + _features.clear_feature(CPU_AVX512_VNNI); + _features.clear_feature(CPU_AVX512_VAES); + _features.clear_feature(CPU_AVX512_VPOPCNTDQ); + _features.clear_feature(CPU_AVX512_VPCLMULQDQ); + _features.clear_feature(CPU_AVX512_VBMI); + _features.clear_feature(CPU_AVX512_VBMI2); + _features.clear_feature(CPU_CLWB); + _features.clear_feature(CPU_FLUSHOPT); + _features.clear_feature(CPU_GFNI); + _features.clear_feature(CPU_AVX512_BITALG); + _features.clear_feature(CPU_AVX512_IFMA); + _features.clear_feature(CPU_AVX_IFMA); + _features.clear_feature(CPU_AVX512_FP16); + _features.clear_feature(CPU_AVX10_1); + _features.clear_feature(CPU_AVX10_2); } } @@ -1089,16 +1077,44 @@ void VM_Version::get_processor_features() { _has_intel_jcc_erratum = IntelJccErratumMitigation; } - char buf[1024]; - int res = jio_snprintf( + assert(supports_clflush(), "Always present"); + if (X86ICacheSync == -1) { + // Auto-detect, choosing the best performant one that still flushes + // the cache. We could switch to CPUID/SERIALIZE ("4"/"5") going forward. + if (supports_clwb()) { + FLAG_SET_ERGO(X86ICacheSync, 3); + } else if (supports_clflushopt()) { + FLAG_SET_ERGO(X86ICacheSync, 2); + } else { + FLAG_SET_ERGO(X86ICacheSync, 1); + } + } else { + if ((X86ICacheSync == 2) && !supports_clflushopt()) { + vm_exit_during_initialization("CPU does not support CLFLUSHOPT, unable to use X86ICacheSync=2"); + } + if ((X86ICacheSync == 3) && !supports_clwb()) { + vm_exit_during_initialization("CPU does not support CLWB, unable to use X86ICacheSync=3"); + } + if ((X86ICacheSync == 5) && !supports_serialize()) { + vm_exit_during_initialization("CPU does not support SERIALIZE, unable to use X86ICacheSync=5"); + } + } + + char buf[2048]; + size_t cpu_info_size = jio_snprintf( buf, sizeof(buf), "(%u cores per cpu, %u threads per core) family %d model %d stepping %d microcode 0x%x", cores_per_cpu(), threads_per_core(), cpu_family(), _model, _stepping, os::cpu_microcode_revision()); - assert(res > 0, "not enough temporary space allocated"); - insert_features_names(buf + res, sizeof(buf) - res, _features_names); + assert(cpu_info_size > 0, "not enough temporary space allocated"); - _features_string = os::strdup(buf); + insert_features_names(_features, buf + cpu_info_size, sizeof(buf) - cpu_info_size); + + _cpu_info_string = os::strdup(buf); + + _features_string = extract_features_string(_cpu_info_string, + strnlen(_cpu_info_string, sizeof(buf)), + cpu_info_size); // Use AES instructions if available. if (supports_aes()) { @@ -1182,7 +1198,6 @@ void VM_Version::get_processor_features() { FLAG_SET_DEFAULT(UseCRC32Intrinsics, false); } -#ifdef _LP64 if (supports_avx2()) { if (FLAG_IS_DEFAULT(UseAdler32Intrinsics)) { UseAdler32Intrinsics = true; @@ -1193,12 +1208,6 @@ void VM_Version::get_processor_features() { } FLAG_SET_DEFAULT(UseAdler32Intrinsics, false); } -#else - if (UseAdler32Intrinsics) { - warning("Adler32Intrinsics not available on this CPU."); - FLAG_SET_DEFAULT(UseAdler32Intrinsics, false); - } -#endif if (supports_sse4_2() && supports_clmul()) { if (FLAG_IS_DEFAULT(UseCRC32CIntrinsics)) { @@ -1222,7 +1231,6 @@ void VM_Version::get_processor_features() { FLAG_SET_DEFAULT(UseGHASHIntrinsics, false); } -#ifdef _LP64 // ChaCha20 Intrinsics // As long as the system supports AVX as a baseline we can do a // SIMD-enabled block function. StubGenerator makes the determination @@ -1238,24 +1246,28 @@ void VM_Version::get_processor_features() { } FLAG_SET_DEFAULT(UseChaCha20Intrinsics, false); } -#else - // No support currently for ChaCha20 intrinsics on 32-bit platforms - if (UseChaCha20Intrinsics) { - warning("ChaCha20 intrinsics are not available on this CPU."); - FLAG_SET_DEFAULT(UseChaCha20Intrinsics, false); + + // Kyber Intrinsics + // Currently we only have them for AVX512 +#ifdef _LP64 + if (supports_evex() && supports_avx512bw()) { + if (FLAG_IS_DEFAULT(UseKyberIntrinsics)) { + UseKyberIntrinsics = true; + } + } else +#endif + if (UseKyberIntrinsics) { + warning("Intrinsics for ML-KEM are not available on this CPU."); + FLAG_SET_DEFAULT(UseKyberIntrinsics, false); } -#endif // _LP64 // Dilithium Intrinsics // Currently we only have them for AVX512 -#ifdef _LP64 if (supports_evex() && supports_avx512bw()) { if (FLAG_IS_DEFAULT(UseDilithiumIntrinsics)) { UseDilithiumIntrinsics = true; } - } else -#endif - if (UseDilithiumIntrinsics) { + } else if (UseDilithiumIntrinsics) { warning("Intrinsics for ML-DSA are not available on this CPU."); FLAG_SET_DEFAULT(UseDilithiumIntrinsics, false); } @@ -1284,7 +1296,7 @@ void VM_Version::get_processor_features() { UseMD5Intrinsics = true; } - if (supports_sha() LP64_ONLY(|| (supports_avx2() && supports_bmi2()))) { + if (supports_sha() || (supports_avx2() && supports_bmi2())) { if (FLAG_IS_DEFAULT(UseSHA)) { UseSHA = true; } @@ -1311,27 +1323,20 @@ void VM_Version::get_processor_features() { FLAG_SET_DEFAULT(UseSHA256Intrinsics, false); } -#ifdef _LP64 - // These are only supported on 64-bit if (UseSHA && supports_avx2() && (supports_bmi2() || supports_sha512())) { if (FLAG_IS_DEFAULT(UseSHA512Intrinsics)) { FLAG_SET_DEFAULT(UseSHA512Intrinsics, true); } - } else -#endif - if (UseSHA512Intrinsics) { + } else if (UseSHA512Intrinsics) { warning("Intrinsics for SHA-384 and SHA-512 crypto hash functions not available on this CPU."); FLAG_SET_DEFAULT(UseSHA512Intrinsics, false); } -#ifdef _LP64 if (supports_evex() && supports_avx512bw()) { if (FLAG_IS_DEFAULT(UseSHA3Intrinsics)) { UseSHA3Intrinsics = true; } - } else -#endif - if (UseSHA3Intrinsics) { + } else if (UseSHA3Intrinsics) { warning("Intrinsics for SHA3-224, SHA3-256, SHA3-384 and SHA3-512 crypto hash functions not available on this CPU."); FLAG_SET_DEFAULT(UseSHA3Intrinsics, false); } @@ -1353,11 +1358,7 @@ void VM_Version::get_processor_features() { max_vector_size = 64; } -#ifdef _LP64 int min_vector_size = 4; // We require MaxVectorSize to be at least 4 on 64bit -#else - int min_vector_size = 0; -#endif if (!FLAG_IS_DEFAULT(MaxVectorSize)) { if (MaxVectorSize < min_vector_size) { @@ -1381,7 +1382,7 @@ void VM_Version::get_processor_features() { if (MaxVectorSize > 0) { if (supports_avx() && PrintMiscellaneous && Verbose && TraceNewVectors) { tty->print_cr("State of YMM registers after signal handle:"); - int nreg = 2 LP64_ONLY(+2); + int nreg = 4; const char* ymm_name[4] = {"0", "7", "8", "15"}; for (int i = 0; i < nreg; i++) { tty->print("YMM%s:", ymm_name[i]); @@ -1394,31 +1395,24 @@ void VM_Version::get_processor_features() { } #endif // COMPILER2 && ASSERT -#ifdef _LP64 if ((supports_avx512ifma() && supports_avx512vlbw()) || supports_avxifma()) { if (FLAG_IS_DEFAULT(UsePoly1305Intrinsics)) { FLAG_SET_DEFAULT(UsePoly1305Intrinsics, true); } - } else -#endif - if (UsePoly1305Intrinsics) { + } else if (UsePoly1305Intrinsics) { warning("Intrinsics for Poly1305 crypto hash functions not available on this CPU."); FLAG_SET_DEFAULT(UsePoly1305Intrinsics, false); } -#ifdef _LP64 if ((supports_avx512ifma() && supports_avx512vlbw()) || supports_avxifma()) { if (FLAG_IS_DEFAULT(UseIntPolyIntrinsics)) { FLAG_SET_DEFAULT(UseIntPolyIntrinsics, true); } - } else -#endif - if (UseIntPolyIntrinsics) { + } else if (UseIntPolyIntrinsics) { warning("Intrinsics for Polynomial crypto functions not available on this CPU."); FLAG_SET_DEFAULT(UseIntPolyIntrinsics, false); } -#ifdef _LP64 if (FLAG_IS_DEFAULT(UseMultiplyToLenIntrinsic)) { UseMultiplyToLenIntrinsic = true; } @@ -1434,38 +1428,6 @@ void VM_Version::get_processor_features() { if (FLAG_IS_DEFAULT(UseMontgomerySquareIntrinsic)) { UseMontgomerySquareIntrinsic = true; } -#else - if (UseMultiplyToLenIntrinsic) { - if (!FLAG_IS_DEFAULT(UseMultiplyToLenIntrinsic)) { - warning("multiplyToLen intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseMultiplyToLenIntrinsic, false); - } - if (UseMontgomeryMultiplyIntrinsic) { - if (!FLAG_IS_DEFAULT(UseMontgomeryMultiplyIntrinsic)) { - warning("montgomeryMultiply intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseMontgomeryMultiplyIntrinsic, false); - } - if (UseMontgomerySquareIntrinsic) { - if (!FLAG_IS_DEFAULT(UseMontgomerySquareIntrinsic)) { - warning("montgomerySquare intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseMontgomerySquareIntrinsic, false); - } - if (UseSquareToLenIntrinsic) { - if (!FLAG_IS_DEFAULT(UseSquareToLenIntrinsic)) { - warning("squareToLen intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseSquareToLenIntrinsic, false); - } - if (UseMulAddIntrinsic) { - if (!FLAG_IS_DEFAULT(UseMulAddIntrinsic)) { - warning("mulAdd intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseMulAddIntrinsic, false); - } -#endif // _LP64 #endif // COMPILER2_OR_JVMCI // On new cpus instructions which update whole XMM register should be used @@ -1632,7 +1594,7 @@ void VM_Version::get_processor_features() { if (FLAG_IS_DEFAULT(UseStoreImmI16)) { UseStoreImmI16 = false; // don't use it on Intel cpus } - if (cpu_family() == 6 || cpu_family() == 15) { + if (is_intel_server_family() || cpu_family() == 15) { if (FLAG_IS_DEFAULT(UseAddressNop)) { // Use it on all Intel cpus starting from PentiumPro UseAddressNop = true; @@ -1648,7 +1610,7 @@ void VM_Version::get_processor_features() { UseXmmRegToRegMoveAll = false; } } - if (cpu_family() == 6 && supports_sse3()) { // New Intel cpus + if (is_intel_server_family() && supports_sse3()) { // New Intel cpus #ifdef COMPILER2 if (FLAG_IS_DEFAULT(MaxLoopPad)) { // For new Intel cpus do the next optimization: @@ -1742,7 +1704,6 @@ void VM_Version::get_processor_features() { } #endif -#ifdef _LP64 if (UseSSE42Intrinsics) { if (FLAG_IS_DEFAULT(UseVectorizedMismatchIntrinsic)) { UseVectorizedMismatchIntrinsic = true; @@ -1759,20 +1720,6 @@ void VM_Version::get_processor_features() { warning("vectorizedHashCode intrinsics are not available on this CPU"); FLAG_SET_DEFAULT(UseVectorizedHashCodeIntrinsic, false); } -#else - if (UseVectorizedMismatchIntrinsic) { - if (!FLAG_IS_DEFAULT(UseVectorizedMismatchIntrinsic)) { - warning("vectorizedMismatch intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseVectorizedMismatchIntrinsic, false); - } - if (UseVectorizedHashCodeIntrinsic) { - if (!FLAG_IS_DEFAULT(UseVectorizedHashCodeIntrinsic)) { - warning("vectorizedHashCode intrinsic is not available in 32-bit VM"); - } - FLAG_SET_DEFAULT(UseVectorizedHashCodeIntrinsic, false); - } -#endif // _LP64 // Use count leading zeros count instruction if available. if (supports_lzcnt()) { @@ -1901,7 +1848,7 @@ void VM_Version::get_processor_features() { FLAG_SET_DEFAULT(AllocatePrefetchDistance, allocate_prefetch_distance(use_watermark_prefetch)); } - if (is_intel() && cpu_family() == 6 && supports_sse3()) { + if (is_intel() && is_intel_server_family() && supports_sse3()) { if (FLAG_IS_DEFAULT(AllocatePrefetchLines) && supports_sse4_2() && supports_ht()) { // Nehalem based cpus FLAG_SET_DEFAULT(AllocatePrefetchLines, 4); @@ -1921,7 +1868,6 @@ void VM_Version::get_processor_features() { #endif } -#ifdef _LP64 // Prefetch settings // Prefetch interval for gc copy/scan == 9 dcache lines. Derived from @@ -1940,7 +1886,6 @@ void VM_Version::get_processor_features() { if (FLAG_IS_DEFAULT(PrefetchScanIntervalInBytes)) { FLAG_SET_DEFAULT(PrefetchScanIntervalInBytes, 576); } -#endif if (FLAG_IS_DEFAULT(ContendedPaddingWidth) && (cache_line_size > ContendedPaddingWidth)) @@ -2171,16 +2116,15 @@ int VM_Version::avx3_threshold() { FLAG_IS_DEFAULT(AVX3Threshold)) ? 0 : AVX3Threshold; } -#if defined(_LP64) void VM_Version::clear_apx_test_state() { clear_apx_test_state_stub(); } -#endif static bool _vm_version_initialized = false; void VM_Version::initialize() { ResourceMark rm; + // Making this stub must be FIRST use of assembler stub_blob = BufferBlob::create("VM_Version stub", stub_size); if (stub_blob == nullptr) { @@ -2193,14 +2137,11 @@ void VM_Version::initialize() { g.generate_get_cpu_info()); detect_virt_stub = CAST_TO_FN_PTR(detect_virt_stub_t, g.generate_detect_virt()); - -#if defined(_LP64) clear_apx_test_state_stub = CAST_TO_FN_PTR(clear_apx_test_state_t, g.clear_apx_test_state()); -#endif get_processor_features(); - LP64_ONLY(Assembler::precompute_instructions();) + Assembler::precompute_instructions(); if (VM_Version::supports_hv()) { // Supports hypervisor check_virtualizations(); @@ -2959,192 +2900,217 @@ int64_t VM_Version::maximum_qualified_cpu_frequency(void) { return _max_qualified_cpu_frequency; } -uint64_t VM_Version::CpuidInfo::feature_flags() const { - uint64_t result = 0; +VM_Version::VM_Features VM_Version::CpuidInfo::feature_flags() const { + VM_Features vm_features; if (std_cpuid1_edx.bits.cmpxchg8 != 0) - result |= CPU_CX8; + vm_features.set_feature(CPU_CX8); if (std_cpuid1_edx.bits.cmov != 0) - result |= CPU_CMOV; + vm_features.set_feature(CPU_CMOV); if (std_cpuid1_edx.bits.clflush != 0) - result |= CPU_FLUSH; -#ifdef _LP64 + vm_features.set_feature(CPU_FLUSH); // clflush should always be available on x86_64 // if not we are in real trouble because we rely on it // to flush the code cache. - assert ((result & CPU_FLUSH) != 0, "clflush should be available"); -#endif + assert (vm_features.supports_feature(CPU_FLUSH), "clflush should be available"); if (std_cpuid1_edx.bits.fxsr != 0 || (is_amd_family() && ext_cpuid1_edx.bits.fxsr != 0)) - result |= CPU_FXSR; + vm_features.set_feature(CPU_FXSR); // HT flag is set for multi-core processors also. if (threads_per_core() > 1) - result |= CPU_HT; + vm_features.set_feature(CPU_HT); if (std_cpuid1_edx.bits.mmx != 0 || (is_amd_family() && ext_cpuid1_edx.bits.mmx != 0)) - result |= CPU_MMX; + vm_features.set_feature(CPU_MMX); if (std_cpuid1_edx.bits.sse != 0) - result |= CPU_SSE; + vm_features.set_feature(CPU_SSE); if (std_cpuid1_edx.bits.sse2 != 0) - result |= CPU_SSE2; + vm_features.set_feature(CPU_SSE2); if (std_cpuid1_ecx.bits.sse3 != 0) - result |= CPU_SSE3; + vm_features.set_feature(CPU_SSE3); if (std_cpuid1_ecx.bits.ssse3 != 0) - result |= CPU_SSSE3; + vm_features.set_feature(CPU_SSSE3); if (std_cpuid1_ecx.bits.sse4_1 != 0) - result |= CPU_SSE4_1; + vm_features.set_feature(CPU_SSE4_1); if (std_cpuid1_ecx.bits.sse4_2 != 0) - result |= CPU_SSE4_2; + vm_features.set_feature(CPU_SSE4_2); if (std_cpuid1_ecx.bits.popcnt != 0) - result |= CPU_POPCNT; + vm_features.set_feature(CPU_POPCNT); if (sefsl1_cpuid7_edx.bits.apx_f != 0 && xem_xcr0_eax.bits.apx_f != 0) { - result |= CPU_APX_F; + vm_features.set_feature(CPU_APX_F); } if (std_cpuid1_ecx.bits.avx != 0 && std_cpuid1_ecx.bits.osxsave != 0 && xem_xcr0_eax.bits.sse != 0 && xem_xcr0_eax.bits.ymm != 0) { - result |= CPU_AVX; - result |= CPU_VZEROUPPER; + vm_features.set_feature(CPU_AVX); + vm_features.set_feature(CPU_VZEROUPPER); if (sefsl1_cpuid7_eax.bits.sha512 != 0) - result |= CPU_SHA512; + vm_features.set_feature(CPU_SHA512); if (std_cpuid1_ecx.bits.f16c != 0) - result |= CPU_F16C; + vm_features.set_feature(CPU_F16C); if (sef_cpuid7_ebx.bits.avx2 != 0) { - result |= CPU_AVX2; + vm_features.set_feature(CPU_AVX2); if (sefsl1_cpuid7_eax.bits.avx_ifma != 0) - result |= CPU_AVX_IFMA; + vm_features.set_feature(CPU_AVX_IFMA); } if (sef_cpuid7_ecx.bits.gfni != 0) - result |= CPU_GFNI; + vm_features.set_feature(CPU_GFNI); if (sef_cpuid7_ebx.bits.avx512f != 0 && xem_xcr0_eax.bits.opmask != 0 && xem_xcr0_eax.bits.zmm512 != 0 && xem_xcr0_eax.bits.zmm32 != 0) { - result |= CPU_AVX512F; + vm_features.set_feature(CPU_AVX512F); if (sef_cpuid7_ebx.bits.avx512cd != 0) - result |= CPU_AVX512CD; + vm_features.set_feature(CPU_AVX512CD); if (sef_cpuid7_ebx.bits.avx512dq != 0) - result |= CPU_AVX512DQ; + vm_features.set_feature(CPU_AVX512DQ); if (sef_cpuid7_ebx.bits.avx512ifma != 0) - result |= CPU_AVX512_IFMA; + vm_features.set_feature(CPU_AVX512_IFMA); if (sef_cpuid7_ebx.bits.avx512pf != 0) - result |= CPU_AVX512PF; + vm_features.set_feature(CPU_AVX512PF); if (sef_cpuid7_ebx.bits.avx512er != 0) - result |= CPU_AVX512ER; + vm_features.set_feature(CPU_AVX512ER); if (sef_cpuid7_ebx.bits.avx512bw != 0) - result |= CPU_AVX512BW; + vm_features.set_feature(CPU_AVX512BW); if (sef_cpuid7_ebx.bits.avx512vl != 0) - result |= CPU_AVX512VL; + vm_features.set_feature(CPU_AVX512VL); if (sef_cpuid7_ecx.bits.avx512_vpopcntdq != 0) - result |= CPU_AVX512_VPOPCNTDQ; + vm_features.set_feature(CPU_AVX512_VPOPCNTDQ); if (sef_cpuid7_ecx.bits.avx512_vpclmulqdq != 0) - result |= CPU_AVX512_VPCLMULQDQ; + vm_features.set_feature(CPU_AVX512_VPCLMULQDQ); if (sef_cpuid7_ecx.bits.vaes != 0) - result |= CPU_AVX512_VAES; + vm_features.set_feature(CPU_AVX512_VAES); if (sef_cpuid7_ecx.bits.avx512_vnni != 0) - result |= CPU_AVX512_VNNI; + vm_features.set_feature(CPU_AVX512_VNNI); if (sef_cpuid7_ecx.bits.avx512_bitalg != 0) - result |= CPU_AVX512_BITALG; + vm_features.set_feature(CPU_AVX512_BITALG); if (sef_cpuid7_ecx.bits.avx512_vbmi != 0) - result |= CPU_AVX512_VBMI; + vm_features.set_feature(CPU_AVX512_VBMI); if (sef_cpuid7_ecx.bits.avx512_vbmi2 != 0) - result |= CPU_AVX512_VBMI2; + vm_features.set_feature(CPU_AVX512_VBMI2); + } + if (is_intel()) { + if (sefsl1_cpuid7_edx.bits.avx10 != 0 && + std_cpuid24_ebx.bits.avx10_vlen_512 !=0 && + std_cpuid24_ebx.bits.avx10_converged_isa_version >= 1 && + xem_xcr0_eax.bits.opmask != 0 && + xem_xcr0_eax.bits.zmm512 != 0 && + xem_xcr0_eax.bits.zmm32 != 0) { + vm_features.set_feature(CPU_AVX10_1); + vm_features.set_feature(CPU_AVX512F); + vm_features.set_feature(CPU_AVX512CD); + vm_features.set_feature(CPU_AVX512DQ); + vm_features.set_feature(CPU_AVX512PF); + vm_features.set_feature(CPU_AVX512ER); + vm_features.set_feature(CPU_AVX512BW); + vm_features.set_feature(CPU_AVX512VL); + vm_features.set_feature(CPU_AVX512_VPOPCNTDQ); + vm_features.set_feature(CPU_AVX512_VPCLMULQDQ); + vm_features.set_feature(CPU_AVX512_VAES); + vm_features.set_feature(CPU_AVX512_VNNI); + vm_features.set_feature(CPU_AVX512_BITALG); + vm_features.set_feature(CPU_AVX512_VBMI); + vm_features.set_feature(CPU_AVX512_VBMI2); + if (std_cpuid24_ebx.bits.avx10_converged_isa_version >= 2) { + vm_features.set_feature(CPU_AVX10_2); + } + } } } + if (std_cpuid1_ecx.bits.hv != 0) - result |= CPU_HV; + vm_features.set_feature(CPU_HV); if (sef_cpuid7_ebx.bits.bmi1 != 0) - result |= CPU_BMI1; + vm_features.set_feature(CPU_BMI1); if (std_cpuid1_edx.bits.tsc != 0) - result |= CPU_TSC; + vm_features.set_feature(CPU_TSC); if (ext_cpuid7_edx.bits.tsc_invariance != 0) - result |= CPU_TSCINV_BIT; + vm_features.set_feature(CPU_TSCINV_BIT); if (std_cpuid1_ecx.bits.aes != 0) - result |= CPU_AES; + vm_features.set_feature(CPU_AES); if (ext_cpuid1_ecx.bits.lzcnt != 0) - result |= CPU_LZCNT; + vm_features.set_feature(CPU_LZCNT); if (ext_cpuid1_ecx.bits.prefetchw != 0) - result |= CPU_3DNOW_PREFETCH; + vm_features.set_feature(CPU_3DNOW_PREFETCH); if (sef_cpuid7_ebx.bits.erms != 0) - result |= CPU_ERMS; + vm_features.set_feature(CPU_ERMS); if (sef_cpuid7_edx.bits.fast_short_rep_mov != 0) - result |= CPU_FSRM; + vm_features.set_feature(CPU_FSRM); if (std_cpuid1_ecx.bits.clmul != 0) - result |= CPU_CLMUL; + vm_features.set_feature(CPU_CLMUL); if (sef_cpuid7_ebx.bits.rtm != 0) - result |= CPU_RTM; + vm_features.set_feature(CPU_RTM); if (sef_cpuid7_ebx.bits.adx != 0) - result |= CPU_ADX; + vm_features.set_feature(CPU_ADX); if (sef_cpuid7_ebx.bits.bmi2 != 0) - result |= CPU_BMI2; + vm_features.set_feature(CPU_BMI2); if (sef_cpuid7_ebx.bits.sha != 0) - result |= CPU_SHA; + vm_features.set_feature(CPU_SHA); if (std_cpuid1_ecx.bits.fma != 0) - result |= CPU_FMA; + vm_features.set_feature(CPU_FMA); if (sef_cpuid7_ebx.bits.clflushopt != 0) - result |= CPU_FLUSHOPT; + vm_features.set_feature(CPU_FLUSHOPT); if (sef_cpuid7_ebx.bits.clwb != 0) - result |= CPU_CLWB; + vm_features.set_feature(CPU_CLWB); if (ext_cpuid1_edx.bits.rdtscp != 0) - result |= CPU_RDTSCP; + vm_features.set_feature(CPU_RDTSCP); if (sef_cpuid7_ecx.bits.rdpid != 0) - result |= CPU_RDPID; + vm_features.set_feature(CPU_RDPID); // AMD|Hygon additional features. if (is_amd_family()) { // PREFETCHW was checked above, check TDNOW here. if ((ext_cpuid1_edx.bits.tdnow != 0)) - result |= CPU_3DNOW_PREFETCH; + vm_features.set_feature(CPU_3DNOW_PREFETCH); if (ext_cpuid1_ecx.bits.sse4a != 0) - result |= CPU_SSE4A; + vm_features.set_feature(CPU_SSE4A); } // Intel additional features. if (is_intel()) { if (sef_cpuid7_edx.bits.serialize != 0) - result |= CPU_SERIALIZE; + vm_features.set_feature(CPU_SERIALIZE); if (_cpuid_info.sef_cpuid7_edx.bits.avx512_fp16 != 0) - result |= CPU_AVX512_FP16; + vm_features.set_feature(CPU_AVX512_FP16); } // ZX additional features. if (is_zx()) { // We do not know if these are supported by ZX, so we cannot trust // common CPUID bit for them. - assert((result & CPU_CLWB) == 0, "Check if it is supported?"); - result &= ~CPU_CLWB; + assert(vm_features.supports_feature(CPU_CLWB), "Check if it is supported?"); + vm_features.clear_feature(CPU_CLWB); } // Protection key features. if (sef_cpuid7_ecx.bits.pku != 0) { - result |= CPU_PKU; + vm_features.set_feature(CPU_PKU); } if (sef_cpuid7_ecx.bits.ospke != 0) { - result |= CPU_OSPKE; + vm_features.set_feature(CPU_OSPKE); } // Control flow enforcement (CET) features. if (sef_cpuid7_ecx.bits.cet_ss != 0) { - result |= CPU_CET_SS; + vm_features.set_feature(CPU_CET_SS); } if (sef_cpuid7_edx.bits.cet_ibt != 0) { - result |= CPU_CET_IBT; + vm_features.set_feature(CPU_CET_IBT); } // Composite features. if (supports_tscinv_bit() && ((is_amd_family() && !is_amd_Barcelona()) || is_intel_tsc_synched_at_init())) { - result |= CPU_TSCINV; + vm_features.set_feature(CPU_TSCINV); } - - return result; + return vm_features; } bool VM_Version::os_supports_avx_vectors() { bool retVal = false; - int nreg = 2 LP64_ONLY(+2); + int nreg = 4; if (supports_evex()) { // Verify that OS save/restore all bits of EVEX registers // during signal processing. @@ -3296,19 +3262,15 @@ int VM_Version::allocate_prefetch_distance(bool use_watermark_prefetch) { return 128; // Athlon } } else { // Intel - if (supports_sse3() && cpu_family() == 6) { + if (supports_sse3() && is_intel_server_family()) { if (supports_sse4_2() && supports_ht()) { // Nehalem based cpus return 192; } else if (use_watermark_prefetch) { // watermark prefetching on Core -#ifdef _LP64 return 384; -#else - return 320; -#endif } } if (supports_sse2()) { - if (cpu_family() == 6) { + if (is_intel_server_family()) { return 256; // Pentium M, Core, Core2 } else { return 512; // Pentium 4 @@ -3333,3 +3295,14 @@ bool VM_Version::is_intrinsic_supported(vmIntrinsicID id) { } return true; } + +void VM_Version::insert_features_names(VM_Version::VM_Features features, char* buf, size_t buflen) { + for (int i = 0; i < MAX_CPU_FEATURES; i++) { + if (features.supports_feature((VM_Version::Feature_Flag)i)) { + int res = jio_snprintf(buf, buflen, ", %s", _features_names[i]); + assert(res > 0, "not enough temporary space allocated"); + buf += res; + buflen -= res; + } + } +} diff --git a/src/hotspot/cpu/x86/vm_version_x86.hpp b/src/hotspot/cpu/x86/vm_version_x86.hpp index cc5c6c1c639..3c8971e474b 100644 --- a/src/hotspot/cpu/x86/vm_version_x86.hpp +++ b/src/hotspot/cpu/x86/vm_version_x86.hpp @@ -295,12 +295,32 @@ class VM_Version : public Abstract_VM_Version { union SefCpuid7SubLeaf1Edx { uint32_t value; struct { - uint32_t : 21, + uint32_t : 19, + avx10 : 1, + : 1, apx_f : 1, : 10; } bits; }; + union StdCpuid24MainLeafEax { + uint32_t value; + struct { + uint32_t sub_leaves_cnt : 31; + } bits; + }; + + union StdCpuid24MainLeafEbx { + uint32_t value; + struct { + uint32_t avx10_converged_isa_version : 8, + : 8, + : 2, + avx10_vlen_512 : 1, + : 13; + } bits; + }; + union ExtCpuid1EEbx { uint32_t value; struct { @@ -342,9 +362,9 @@ class VM_Version : public Abstract_VM_Version { /* * Update following files when declaring new flags: * test/lib-test/jdk/test/whitebox/CPUInfoTest.java - * src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.amd64/src/jdk/vm/ci/amd64/AMD64.java + * src/jdk.internal.vm.ci/share/classes/jdk/vm/ci/amd64/AMD64.java */ - enum Feature_Flag : uint64_t { + enum Feature_Flag { #define CPU_FEATURE_FLAGS(decl) \ decl(CX8, "cx8", 0) /* next bits are from cpuid 1 (EDX) */ \ decl(CMOV, "cmov", 1) \ @@ -420,15 +440,85 @@ class VM_Version : public Abstract_VM_Version { decl(AVX_IFMA, "avx_ifma", 59) /* 256-bit VEX-coded variant of AVX512-IFMA*/ \ decl(APX_F, "apx_f", 60) /* Intel Advanced Performance Extensions*/ \ decl(SHA512, "sha512", 61) /* SHA512 instructions*/ \ - decl(AVX512_FP16, "avx512_fp16", 62) /* AVX512 FP16 ISA support*/ + decl(AVX512_FP16, "avx512_fp16", 62) /* AVX512 FP16 ISA support*/ \ + decl(AVX10_1, "avx10_1", 63) /* AVX10 512 bit vector ISA Version 1 support*/ \ + decl(AVX10_2, "avx10_2", 64) /* AVX10 512 bit vector ISA Version 2 support*/ -#define DECLARE_CPU_FEATURE_FLAG(id, name, bit) CPU_##id = (1ULL << bit), +#define DECLARE_CPU_FEATURE_FLAG(id, name, bit) CPU_##id = (bit), CPU_FEATURE_FLAGS(DECLARE_CPU_FEATURE_FLAG) #undef DECLARE_CPU_FEATURE_FLAG + MAX_CPU_FEATURES }; + class VM_Features { + friend class VMStructs; + friend class JVMCIVMStructs; + + private: + uint64_t _features_bitmap[(MAX_CPU_FEATURES / BitsPerLong) + 1]; + + STATIC_ASSERT(sizeof(_features_bitmap) * BitsPerByte >= MAX_CPU_FEATURES); + + // Number of 8-byte elements in _bitmap. + constexpr static int features_bitmap_element_count() { + return sizeof(_features_bitmap) / sizeof(uint64_t); + } + + constexpr static int features_bitmap_element_shift_count() { + return LogBitsPerLong; + } + + constexpr static uint64_t features_bitmap_element_mask() { + return (1ULL << features_bitmap_element_shift_count()) - 1; + } + + static int index(Feature_Flag feature) { + int idx = feature >> features_bitmap_element_shift_count(); + assert(idx < features_bitmap_element_count(), "Features array index out of bounds"); + return idx; + } + + static uint64_t bit_mask(Feature_Flag feature) { + return (1ULL << (feature & features_bitmap_element_mask())); + } + + static int _features_bitmap_size; // for JVMCI purposes + public: + VM_Features() { + for (int i = 0; i < features_bitmap_element_count(); i++) { + _features_bitmap[i] = 0; + } + } + + void set_feature(Feature_Flag feature) { + int idx = index(feature); + _features_bitmap[idx] |= bit_mask(feature); + } + + void clear_feature(VM_Version::Feature_Flag feature) { + int idx = index(feature); + _features_bitmap[idx] &= ~bit_mask(feature); + } + + bool supports_feature(VM_Version::Feature_Flag feature) { + int idx = index(feature); + return (_features_bitmap[idx] & bit_mask(feature)) != 0; + } + }; + + // CPU feature flags vector, can be affected by VM settings. + static VM_Features _features; + + // Original CPU feature flags vector, not affected by VM settings. + static VM_Features _cpu_features; + static const char* _features_names[]; + static void clear_cpu_features() { + _features = VM_Features(); + _cpu_features = VM_Features(); + } + enum Extended_Family { // AMD CPU_FAMILY_AMD_11H = 0x11, @@ -492,6 +582,11 @@ class VM_Version : public Abstract_VM_Version { SefCpuid7SubLeaf1Eax sefsl1_cpuid7_eax; SefCpuid7SubLeaf1Edx sefsl1_cpuid7_edx; + // cpuid function 24 converged vector ISA main leaf + // eax = 24, ecx = 0 + StdCpuid24MainLeafEax std_cpuid24_eax; + StdCpuid24MainLeafEbx std_cpuid24_ebx; + // cpuid function 0xB (processor topology) // ecx = 0 uint32_t tpl_cpuidB0_eax; @@ -565,7 +660,7 @@ class VM_Version : public Abstract_VM_Version { // Space to save apx registers after signal handle jlong apx_save[2]; // Save r16 and r31 - uint64_t feature_flags() const; + VM_Features feature_flags() const; // Asserts void assert_is_initialized() const { @@ -611,6 +706,7 @@ class VM_Version : public Abstract_VM_Version { // Offsets for cpuid asm stub static ByteSize std_cpuid0_offset() { return byte_offset_of(CpuidInfo, std_max_function); } static ByteSize std_cpuid1_offset() { return byte_offset_of(CpuidInfo, std_cpuid1_eax); } + static ByteSize std_cpuid24_offset() { return byte_offset_of(CpuidInfo, std_cpuid24_eax); } static ByteSize dcp_cpuid4_offset() { return byte_offset_of(CpuidInfo, dcp_cpuid4_eax); } static ByteSize sef_cpuid7_offset() { return byte_offset_of(CpuidInfo, sef_cpuid7_eax); } static ByteSize sefsl1_cpuid7_offset() { return byte_offset_of(CpuidInfo, sefsl1_cpuid7_eax); } @@ -642,13 +738,31 @@ class VM_Version : public Abstract_VM_Version { static void set_cpuinfo_cont_addr_apx(address pc) { _cpuinfo_cont_addr_apx = pc; } static address cpuinfo_cont_addr_apx() { return _cpuinfo_cont_addr_apx; } - LP64_ONLY(static void clear_apx_test_state()); + static void clear_apx_test_state(); - static void clean_cpuFeatures() { _features = 0; } - static void set_avx_cpuFeatures() { _features |= (CPU_SSE | CPU_SSE2 | CPU_AVX | CPU_VZEROUPPER ); } - static void set_evex_cpuFeatures() { _features |= (CPU_AVX512F | CPU_SSE | CPU_SSE2 | CPU_VZEROUPPER ); } - static void set_apx_cpuFeatures() { _features |= CPU_APX_F; } - static void set_bmi_cpuFeatures() { _features |= (CPU_BMI1 | CPU_BMI2 | CPU_LZCNT | CPU_POPCNT); } + static void clean_cpuFeatures() { + VM_Version::clear_cpu_features(); + } + static void set_avx_cpuFeatures() { + _features.set_feature(CPU_SSE); + _features.set_feature(CPU_SSE2); + _features.set_feature(CPU_AVX); + _features.set_feature(CPU_VZEROUPPER); + } + static void set_evex_cpuFeatures() { + _features.set_feature(CPU_AVX10_1); + _features.set_feature(CPU_AVX512F); + _features.set_feature(CPU_SSE); + _features.set_feature(CPU_SSE2); + _features.set_feature(CPU_VZEROUPPER); + } + static void set_apx_cpuFeatures() { _features.set_feature(CPU_APX_F); } + static void set_bmi_cpuFeatures() { + _features.set_feature(CPU_BMI1); + _features.set_feature(CPU_BMI2); + _features.set_feature(CPU_LZCNT); + _features.set_feature(CPU_POPCNT); + } // Initialization static void initialize(); @@ -677,6 +791,7 @@ class VM_Version : public Abstract_VM_Version { static uint32_t cpu_stepping() { return _cpuid_info.cpu_stepping(); } static int cpu_family() { return _cpu;} static bool is_P6() { return cpu_family() >= 6; } + static bool is_intel_server_family() { return cpu_family() == 6 || cpu_family() == 19; } static bool is_amd() { assert_is_initialized(); return _cpuid_info.std_vendor_name_0 == 0x68747541; } // 'htuA' static bool is_hygon() { assert_is_initialized(); return _cpuid_info.std_vendor_name_0 == 0x6F677948; } // 'ogyH' static bool is_amd_family() { return is_amd() || is_hygon(); } @@ -703,40 +818,39 @@ class VM_Version : public Abstract_VM_Version { // // Feature identification which can be affected by VM settings // - static bool supports_cpuid() { return _features != 0; } - static bool supports_cmov() { return (_features & CPU_CMOV) != 0; } - static bool supports_fxsr() { return (_features & CPU_FXSR) != 0; } - static bool supports_ht() { return (_features & CPU_HT) != 0; } - static bool supports_mmx() { return (_features & CPU_MMX) != 0; } - static bool supports_sse() { return (_features & CPU_SSE) != 0; } - static bool supports_sse2() { return (_features & CPU_SSE2) != 0; } - static bool supports_sse3() { return (_features & CPU_SSE3) != 0; } - static bool supports_ssse3() { return (_features & CPU_SSSE3)!= 0; } - static bool supports_sse4_1() { return (_features & CPU_SSE4_1) != 0; } - static bool supports_sse4_2() { return (_features & CPU_SSE4_2) != 0; } - static bool supports_popcnt() { return (_features & CPU_POPCNT) != 0; } - static bool supports_avx() { return (_features & CPU_AVX) != 0; } - static bool supports_avx2() { return (_features & CPU_AVX2) != 0; } - static bool supports_tsc() { return (_features & CPU_TSC) != 0; } - static bool supports_rdtscp() { return (_features & CPU_RDTSCP) != 0; } - static bool supports_rdpid() { return (_features & CPU_RDPID) != 0; } - static bool supports_aes() { return (_features & CPU_AES) != 0; } - static bool supports_erms() { return (_features & CPU_ERMS) != 0; } - static bool supports_fsrm() { return (_features & CPU_FSRM) != 0; } - static bool supports_clmul() { return (_features & CPU_CLMUL) != 0; } - static bool supports_rtm() { return (_features & CPU_RTM) != 0; } - static bool supports_bmi1() { return (_features & CPU_BMI1) != 0; } - static bool supports_bmi2() { return (_features & CPU_BMI2) != 0; } - static bool supports_adx() { return (_features & CPU_ADX) != 0; } - static bool supports_evex() { return (_features & CPU_AVX512F) != 0; } - static bool supports_avx512dq() { return (_features & CPU_AVX512DQ) != 0; } - static bool supports_avx512ifma() { return (_features & CPU_AVX512_IFMA) != 0; } - static bool supports_avxifma() { return (_features & CPU_AVX_IFMA) != 0; } - static bool supports_avx512pf() { return (_features & CPU_AVX512PF) != 0; } - static bool supports_avx512er() { return (_features & CPU_AVX512ER) != 0; } - static bool supports_avx512cd() { return (_features & CPU_AVX512CD) != 0; } - static bool supports_avx512bw() { return (_features & CPU_AVX512BW) != 0; } - static bool supports_avx512vl() { return (_features & CPU_AVX512VL) != 0; } + static bool supports_cmov() { return _features.supports_feature(CPU_CMOV); } + static bool supports_fxsr() { return _features.supports_feature(CPU_FXSR); } + static bool supports_ht() { return _features.supports_feature(CPU_HT); } + static bool supports_mmx() { return _features.supports_feature(CPU_MMX); } + static bool supports_sse() { return _features.supports_feature(CPU_SSE); } + static bool supports_sse2() { return _features.supports_feature(CPU_SSE2); } + static bool supports_sse3() { return _features.supports_feature(CPU_SSE3); } + static bool supports_ssse3() { return _features.supports_feature(CPU_SSSE3); } + static bool supports_sse4_1() { return _features.supports_feature(CPU_SSE4_1); } + static bool supports_sse4_2() { return _features.supports_feature(CPU_SSE4_2); } + static bool supports_popcnt() { return _features.supports_feature(CPU_POPCNT); } + static bool supports_avx() { return _features.supports_feature(CPU_AVX); } + static bool supports_avx2() { return _features.supports_feature(CPU_AVX2); } + static bool supports_tsc() { return _features.supports_feature(CPU_TSC); } + static bool supports_rdtscp() { return _features.supports_feature(CPU_RDTSCP); } + static bool supports_rdpid() { return _features.supports_feature(CPU_RDPID); } + static bool supports_aes() { return _features.supports_feature(CPU_AES); } + static bool supports_erms() { return _features.supports_feature(CPU_ERMS); } + static bool supports_fsrm() { return _features.supports_feature(CPU_FSRM); } + static bool supports_clmul() { return _features.supports_feature(CPU_CLMUL); } + static bool supports_rtm() { return _features.supports_feature(CPU_RTM); } + static bool supports_bmi1() { return _features.supports_feature(CPU_BMI1); } + static bool supports_bmi2() { return _features.supports_feature(CPU_BMI2); } + static bool supports_adx() { return _features.supports_feature(CPU_ADX); } + static bool supports_evex() { return _features.supports_feature(CPU_AVX512F); } + static bool supports_avx512dq() { return _features.supports_feature(CPU_AVX512DQ); } + static bool supports_avx512ifma() { return _features.supports_feature(CPU_AVX512_IFMA); } + static bool supports_avxifma() { return _features.supports_feature(CPU_AVX_IFMA); } + static bool supports_avx512pf() { return _features.supports_feature(CPU_AVX512PF); } + static bool supports_avx512er() { return _features.supports_feature(CPU_AVX512ER); } + static bool supports_avx512cd() { return _features.supports_feature(CPU_AVX512CD); } + static bool supports_avx512bw() { return _features.supports_feature(CPU_AVX512BW); } + static bool supports_avx512vl() { return _features.supports_feature(CPU_AVX512VL); } static bool supports_avx512vlbw() { return (supports_evex() && supports_avx512bw() && supports_avx512vl()); } static bool supports_avx512bwdq() { return (supports_evex() && supports_avx512bw() && supports_avx512dq()); } static bool supports_avx512vldq() { return (supports_evex() && supports_avx512dq() && supports_avx512vl()); } @@ -745,33 +859,39 @@ class VM_Version : public Abstract_VM_Version { static bool supports_avx512novl() { return (supports_evex() && !supports_avx512vl()); } static bool supports_avx512nobw() { return (supports_evex() && !supports_avx512bw()); } static bool supports_avx256only() { return (supports_avx2() && !supports_evex()); } - static bool supports_apx_f() { return (_features & CPU_APX_F) != 0; } + static bool supports_apx_f() { return _features.supports_feature(CPU_APX_F); } static bool supports_avxonly() { return ((supports_avx2() || supports_avx()) && !supports_evex()); } - static bool supports_sha() { return (_features & CPU_SHA) != 0; } - static bool supports_fma() { return (_features & CPU_FMA) != 0 && supports_avx(); } - static bool supports_vzeroupper() { return (_features & CPU_VZEROUPPER) != 0; } - static bool supports_avx512_vpopcntdq() { return (_features & CPU_AVX512_VPOPCNTDQ) != 0; } - static bool supports_avx512_vpclmulqdq() { return (_features & CPU_AVX512_VPCLMULQDQ) != 0; } - static bool supports_avx512_vaes() { return (_features & CPU_AVX512_VAES) != 0; } - static bool supports_gfni() { return (_features & CPU_GFNI) != 0; } - static bool supports_avx512_vnni() { return (_features & CPU_AVX512_VNNI) != 0; } - static bool supports_avx512_bitalg() { return (_features & CPU_AVX512_BITALG) != 0; } - static bool supports_avx512_vbmi() { return (_features & CPU_AVX512_VBMI) != 0; } - static bool supports_avx512_vbmi2() { return (_features & CPU_AVX512_VBMI2) != 0; } - static bool supports_avx512_fp16() { return (_features & CPU_AVX512_FP16) != 0; } - static bool supports_hv() { return (_features & CPU_HV) != 0; } - static bool supports_serialize() { return (_features & CPU_SERIALIZE) != 0; } - static bool supports_f16c() { return (_features & CPU_F16C) != 0; } - static bool supports_pku() { return (_features & CPU_PKU) != 0; } - static bool supports_ospke() { return (_features & CPU_OSPKE) != 0; } - static bool supports_cet_ss() { return (_features & CPU_CET_SS) != 0; } - static bool supports_cet_ibt() { return (_features & CPU_CET_IBT) != 0; } - static bool supports_sha512() { return (_features & CPU_SHA512) != 0; } + static bool supports_sha() { return _features.supports_feature(CPU_SHA); } + static bool supports_fma() { return _features.supports_feature(CPU_FMA) && supports_avx(); } + static bool supports_vzeroupper() { return _features.supports_feature(CPU_VZEROUPPER); } + static bool supports_avx512_vpopcntdq() { return _features.supports_feature(CPU_AVX512_VPOPCNTDQ); } + static bool supports_avx512_vpclmulqdq() { return _features.supports_feature(CPU_AVX512_VPCLMULQDQ); } + static bool supports_avx512_vaes() { return _features.supports_feature(CPU_AVX512_VAES); } + static bool supports_gfni() { return _features.supports_feature(CPU_GFNI); } + static bool supports_avx512_vnni() { return _features.supports_feature(CPU_AVX512_VNNI); } + static bool supports_avx512_bitalg() { return _features.supports_feature(CPU_AVX512_BITALG); } + static bool supports_avx512_vbmi() { return _features.supports_feature(CPU_AVX512_VBMI); } + static bool supports_avx512_vbmi2() { return _features.supports_feature(CPU_AVX512_VBMI2); } + static bool supports_avx512_fp16() { return _features.supports_feature(CPU_AVX512_FP16); } + static bool supports_hv() { return _features.supports_feature(CPU_HV); } + static bool supports_serialize() { return _features.supports_feature(CPU_SERIALIZE); } + static bool supports_f16c() { return _features.supports_feature(CPU_F16C); } + static bool supports_pku() { return _features.supports_feature(CPU_PKU); } + static bool supports_ospke() { return _features.supports_feature(CPU_OSPKE); } + static bool supports_cet_ss() { return _features.supports_feature(CPU_CET_SS); } + static bool supports_cet_ibt() { return _features.supports_feature(CPU_CET_IBT); } + static bool supports_sha512() { return _features.supports_feature(CPU_SHA512); } + + // Intel® AVX10 introduces a versioned approach for enumeration that is monotonically increasing, inclusive, + // and supporting all vector lengths. Feature set supported by an AVX10 vector ISA version is also supported + // by all the versions above it. + static bool supports_avx10_1() { return _features.supports_feature(CPU_AVX10_1);} + static bool supports_avx10_2() { return _features.supports_feature(CPU_AVX10_2);} // // Feature identification not affected by VM flags // - static bool cpu_supports_evex() { return (_cpu_features & CPU_AVX512F) != 0; } + static bool cpu_supports_evex() { return _cpu_features.supports_feature(CPU_AVX512F); } static bool supports_avx512_simd_sort() { if (supports_avx512dq()) { @@ -802,6 +922,8 @@ class VM_Version : public Abstract_VM_Version { static bool is_intel_tsc_synched_at_init(); + static void insert_features_names(VM_Version::VM_Features features, char* buf, size_t buflen); + // This checks if the JVM is potentially affected by an erratum on Intel CPUs (SKX102) // that causes unpredictable behaviour when jcc crosses 64 byte boundaries. Its microcode // mitigation causes regressions when jumps or fused conditional branches cross or end at @@ -809,23 +931,23 @@ class VM_Version : public Abstract_VM_Version { static bool has_intel_jcc_erratum() { return _has_intel_jcc_erratum; } // AMD features - static bool supports_3dnow_prefetch() { return (_features & CPU_3DNOW_PREFETCH) != 0; } - static bool supports_lzcnt() { return (_features & CPU_LZCNT) != 0; } - static bool supports_sse4a() { return (_features & CPU_SSE4A) != 0; } + static bool supports_3dnow_prefetch() { return _features.supports_feature(CPU_3DNOW_PREFETCH); } + static bool supports_lzcnt() { return _features.supports_feature(CPU_LZCNT); } + static bool supports_sse4a() { return _features.supports_feature(CPU_SSE4A); } static bool is_amd_Barcelona() { return is_amd() && extended_cpu_family() == CPU_FAMILY_AMD_11H; } // Intel and AMD newer cores support fast timestamps well static bool supports_tscinv_bit() { - return (_features & CPU_TSCINV_BIT) != 0; + return _features.supports_feature(CPU_TSCINV_BIT); } static bool supports_tscinv() { - return (_features & CPU_TSCINV) != 0; + return _features.supports_feature(CPU_TSCINV); } // Intel Core and newer cpus have fast IDIV instruction (excluding Atom). - static bool has_fast_idiv() { return is_intel() && cpu_family() == 6 && + static bool has_fast_idiv() { return is_intel() && is_intel_server_family() && supports_sse3() && _model != 0x1C; } static bool supports_compare_and_exchange() { return true; } @@ -839,12 +961,12 @@ class VM_Version : public Abstract_VM_Version { // x86_64 supports fast class initialization checks static bool supports_fast_class_init_checks() { - return LP64_ONLY(true) NOT_LP64(false); // not implemented on x86_32 + return true; } // x86_64 supports secondary supers table constexpr static bool supports_secondary_supers_table() { - return LP64_ONLY(true) NOT_LP64(false); // not implemented on x86_32 + return true; } constexpr static bool supports_stack_watermark_barrier() { @@ -879,15 +1001,11 @@ class VM_Version : public Abstract_VM_Version { // synchronize with other memory ops. so, it needs preceding // and trailing StoreStore fences. -#ifdef _LP64 static bool supports_clflush(); // Can't inline due to header file conflict -#else - static bool supports_clflush() { return ((_features & CPU_FLUSH) != 0); } -#endif // _LP64 // Note: CPU_FLUSHOPT and CPU_CLWB bits should always be zero for 32-bit - static bool supports_clflushopt() { return ((_features & CPU_FLUSHOPT) != 0); } - static bool supports_clwb() { return ((_features & CPU_CLWB) != 0); } + static bool supports_clflushopt() { return (_features.supports_feature(CPU_FLUSHOPT)); } + static bool supports_clwb() { return (_features.supports_feature(CPU_CLWB)); } // Old CPUs perform lea on AGU which causes additional latency transferring the // value from/to ALU for other operations diff --git a/src/hotspot/cpu/x86/x86_64.ad b/src/hotspot/cpu/x86/x86_64.ad index 078150c61fb..22490ba7bb3 100644 --- a/src/hotspot/cpu/x86/x86_64.ad +++ b/src/hotspot/cpu/x86/x86_64.ad @@ -422,6 +422,18 @@ source_hpp %{ #include "peephole_x86_64.hpp" +bool castLL_is_imm32(const Node* n); + +%} + +source %{ + +bool castLL_is_imm32(const Node* n) { + assert(n->is_CastLL(), "must be a CastLL"); + const TypeLong* t = n->bottom_type()->is_long(); + return (t->_lo == min_jlong || Assembler::is_simm32(t->_lo)) && (t->_hi == max_jlong || Assembler::is_simm32(t->_hi)); +} + %} // Register masks @@ -1584,14 +1596,11 @@ uint MachUEPNode::size(PhaseRegAlloc* ra_) const //============================================================================= bool Matcher::supports_vector_calling_convention(void) { - if (EnableVectorSupport && UseVectorStubs) { - return true; - } - return false; + return EnableVectorSupport; } OptoRegPair Matcher::vector_return_value(uint ideal_reg) { - assert(EnableVectorSupport && UseVectorStubs, "sanity"); + assert(EnableVectorSupport, "sanity"); int lo = XMM0_num; int hi = XMM0b_num; if (ideal_reg == Op_VecX) hi = XMM0d_num; @@ -1838,20 +1847,19 @@ encode %{ %} enc_class clear_avx %{ - debug_only(int off0 = __ offset()); + DEBUG_ONLY(int off0 = __ offset()); if (generate_vzeroupper(Compile::current())) { // Clear upper bits of YMM registers to avoid AVX <-> SSE transition penalty // Clear upper bits of YMM registers when current compiled code uses // wide vectors to avoid AVX <-> SSE transition penalty during call. __ vzeroupper(); } - debug_only(int off1 = __ offset()); + DEBUG_ONLY(int off1 = __ offset()); assert(off1 - off0 == clear_avx_size(), "correct size prediction"); %} enc_class Java_To_Runtime(method meth) %{ - // No relocation needed - __ mov64(r10, (int64_t) $meth$$method); + __ lea(r10, RuntimeAddress((address)$meth$$method)); __ call(r10); __ post_call_nop(); %} @@ -6258,13 +6266,14 @@ instruct cmovI_regUCF2_ne(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegI dst, rRegI src) instruct cmovI_regUCF2_ne_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegI dst, rRegI src1, rRegI src2) %{ predicate(UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::ne); match(Set dst (CMoveI (Binary cop cr) (Binary src1 src2))); + effect(TEMP dst); ins_cost(200); format %{ "ecmovpl $dst, $src1, $src2\n\t" - "ecmovnel $dst, $src1, $src2" %} + "cmovnel $dst, $src2" %} ins_encode %{ __ ecmovl(Assembler::parity, $dst$$Register, $src1$$Register, $src2$$Register); - __ ecmovl(Assembler::notEqual, $dst$$Register, $src1$$Register, $src2$$Register); + __ cmovl(Assembler::notEqual, $dst$$Register, $src2$$Register); %} ins_pipe(pipe_cmov_reg); %} @@ -6274,6 +6283,7 @@ instruct cmovI_regUCF2_ne_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegI dst, rRegI s instruct cmovI_regUCF2_eq(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegI dst, rRegI src) %{ predicate(!UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::eq); match(Set dst (CMoveI (Binary cop cr) (Binary src dst))); + effect(TEMP dst); ins_cost(200); // XXX format %{ "cmovpl $dst, $src\n\t" @@ -6289,14 +6299,15 @@ instruct cmovI_regUCF2_eq(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegI dst, rRegI src) // and parity flag bit is set if any of the operand is a NaN. instruct cmovI_regUCF2_eq_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegI dst, rRegI src1, rRegI src2) %{ predicate(UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::eq); - match(Set dst (CMoveI (Binary cop cr) (Binary src1 src2))); + match(Set dst (CMoveI (Binary cop cr) (Binary src2 src1))); + effect(TEMP dst); ins_cost(200); format %{ "ecmovpl $dst, $src1, $src2\n\t" - "ecmovnel $dst, $src1, $src2" %} + "cmovnel $dst, $src2" %} ins_encode %{ __ ecmovl(Assembler::parity, $dst$$Register, $src1$$Register, $src2$$Register); - __ ecmovl(Assembler::notEqual, $dst$$Register, $src1$$Register, $src2$$Register); + __ cmovl(Assembler::notEqual, $dst$$Register, $src2$$Register); %} ins_pipe(pipe_cmov_reg); %} @@ -6539,6 +6550,7 @@ instruct cmovP_regU_ndd(rRegP dst, cmpOpU cop, rFlagsRegU cr, rRegP src1, rRegP %} instruct cmovP_regUCF(cmpOpUCF cop, rFlagsRegUCF cr, rRegP dst, rRegP src) %{ + predicate(!UseAPX); match(Set dst (CMoveP (Binary cop cr) (Binary dst src))); ins_cost(200); expand %{ @@ -6547,6 +6559,7 @@ instruct cmovP_regUCF(cmpOpUCF cop, rFlagsRegUCF cr, rRegP dst, rRegP src) %{ %} instruct cmovP_regUCF_ndd(rRegP dst, cmpOpUCF cop, rFlagsRegUCF cr, rRegP src1, rRegP src2) %{ + predicate(UseAPX); match(Set dst (CMoveP (Binary cop cr) (Binary src1 src2))); ins_cost(200); format %{ "ecmovq$cop $dst, $src1, $src2\t# unsigned, ptr ndd" %} @@ -6573,13 +6586,14 @@ instruct cmovP_regUCF2_ne(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegP dst, rRegP src) instruct cmovP_regUCF2_ne_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegP dst, rRegP src1, rRegP src2) %{ predicate(UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::ne); match(Set dst (CMoveP (Binary cop cr) (Binary src1 src2))); + effect(TEMP dst); ins_cost(200); format %{ "ecmovpq $dst, $src1, $src2\n\t" - "ecmovneq $dst, $src1, $src2" %} + "cmovneq $dst, $src2" %} ins_encode %{ __ ecmovq(Assembler::parity, $dst$$Register, $src1$$Register, $src2$$Register); - __ ecmovq(Assembler::notEqual, $dst$$Register, $src1$$Register, $src2$$Register); + __ cmovq(Assembler::notEqual, $dst$$Register, $src2$$Register); %} ins_pipe(pipe_cmov_reg); %} @@ -6602,14 +6616,15 @@ instruct cmovP_regUCF2_eq(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegP dst, rRegP src) instruct cmovP_regUCF2_eq_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegP dst, rRegP src1, rRegP src2) %{ predicate(UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::eq); - match(Set dst (CMoveP (Binary cop cr) (Binary src1 src2))); + match(Set dst (CMoveP (Binary cop cr) (Binary src2 src1))); + effect(TEMP dst); ins_cost(200); format %{ "ecmovpq $dst, $src1, $src2\n\t" - "ecmovneq $dst, $src1, $src2" %} + "cmovneq $dst, $src2" %} ins_encode %{ __ ecmovq(Assembler::parity, $dst$$Register, $src1$$Register, $src2$$Register); - __ ecmovq(Assembler::notEqual, $dst$$Register, $src1$$Register, $src2$$Register); + __ cmovq(Assembler::notEqual, $dst$$Register, $src2$$Register); %} ins_pipe(pipe_cmov_reg); %} @@ -6772,13 +6787,14 @@ instruct cmovL_regUCF2_ne(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegL dst, rRegL src) instruct cmovL_regUCF2_ne_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegL dst, rRegL src1, rRegL src2) %{ predicate(UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::ne); match(Set dst (CMoveL (Binary cop cr) (Binary src1 src2))); + effect(TEMP dst); ins_cost(200); format %{ "ecmovpq $dst, $src1, $src2\n\t" - "ecmovneq $dst, $src1, $src2" %} + "cmovneq $dst, $src2" %} ins_encode %{ __ ecmovq(Assembler::parity, $dst$$Register, $src1$$Register, $src2$$Register); - __ ecmovq(Assembler::notEqual, $dst$$Register, $src1$$Register, $src2$$Register); + __ cmovq(Assembler::notEqual, $dst$$Register, $src2$$Register); %} ins_pipe(pipe_cmov_reg); %} @@ -6801,14 +6817,15 @@ instruct cmovL_regUCF2_eq(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegL dst, rRegL src) instruct cmovL_regUCF2_eq_ndd(cmpOpUCF2 cop, rFlagsRegUCF cr, rRegL dst, rRegL src1, rRegL src2) %{ predicate(UseAPX && n->in(1)->in(1)->as_Bool()->_test._test == BoolTest::eq); - match(Set dst (CMoveL (Binary cop cr) (Binary src1 src2))); + match(Set dst (CMoveL (Binary cop cr) (Binary src2 src1))); + effect(TEMP dst); ins_cost(200); format %{ "ecmovpq $dst, $src1, $src2\n\t" - "ecmovneq $dst, $src1, $src2" %} + "cmovneq $dst, $src2" %} ins_encode %{ __ ecmovq(Assembler::parity, $dst$$Register, $src1$$Register, $src2$$Register); - __ ecmovq(Assembler::notEqual, $dst$$Register, $src1$$Register, $src2$$Register); + __ cmovq(Assembler::notEqual, $dst$$Register, $src2$$Register); %} ins_pipe(pipe_cmov_reg); %} @@ -7035,21 +7052,6 @@ instruct addI_rReg_mem(rRegI dst, memory src, rFlagsReg cr) ins_pipe(ialu_reg_mem); %} -instruct addI_rReg_mem_rReg_ndd(rRegI dst, memory src1, rRegI src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (AddI (LoadI src1) src2)); - effect(KILL cr); - flag(PD::Flag_sets_overflow_flag, PD::Flag_sets_sign_flag, PD::Flag_sets_zero_flag, PD::Flag_sets_carry_flag, PD::Flag_sets_parity_flag); - - ins_cost(150); - format %{ "eaddl $dst, $src1, $src2\t# int ndd" %} - ins_encode %{ - __ eaddl($dst$$Register, $src1$$Address, $src2$$Register, false); - %} - ins_pipe(ialu_reg_mem); -%} - instruct addI_rReg_rReg_mem_ndd(rRegI dst, rRegI src1, memory src2, rFlagsReg cr) %{ predicate(UseAPX); @@ -7353,21 +7355,6 @@ instruct addL_rReg_rReg_mem_ndd(rRegL dst, rRegL src1, memory src2, rFlagsReg cr ins_pipe(ialu_reg_mem); %} -instruct addL_rReg_mem_rReg_ndd(rRegL dst, memory src1, rRegL src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (AddL (LoadL src1) src2)); - effect(KILL cr); - flag(PD::Flag_sets_overflow_flag, PD::Flag_sets_sign_flag, PD::Flag_sets_zero_flag, PD::Flag_sets_carry_flag, PD::Flag_sets_parity_flag); - - ins_cost(150); - format %{ "eaddq $dst, $src1, $src2\t# long ndd" %} - ins_encode %{ - __ eaddq($dst$$Register, $src1$$Address, $src2$$Register, false); - %} - ins_pipe(ialu_reg_mem); -%} - instruct addL_mem_rReg(memory dst, rRegL src, rFlagsReg cr) %{ match(Set dst (StoreL dst (AddL (LoadL dst) src))); @@ -7605,6 +7592,7 @@ instruct castPP(rRegP dst) instruct castII(rRegI dst) %{ + predicate(VerifyConstraintCasts == 0); match(Set dst (CastII dst)); size(0); @@ -7614,8 +7602,22 @@ instruct castII(rRegI dst) ins_pipe(empty); %} +instruct castII_checked(rRegI dst, rFlagsReg cr) +%{ + predicate(VerifyConstraintCasts > 0); + match(Set dst (CastII dst)); + + effect(KILL cr); + format %{ "# cast_checked_II $dst" %} + ins_encode %{ + __ verify_int_in_range(_idx, bottom_type()->is_int(), $dst$$Register); + %} + ins_pipe(pipe_slow); +%} + instruct castLL(rRegL dst) %{ + predicate(VerifyConstraintCasts == 0); match(Set dst (CastLL dst)); size(0); @@ -7625,6 +7627,32 @@ instruct castLL(rRegL dst) ins_pipe(empty); %} +instruct castLL_checked_L32(rRegL dst, rFlagsReg cr) +%{ + predicate(VerifyConstraintCasts > 0 && castLL_is_imm32(n)); + match(Set dst (CastLL dst)); + + effect(KILL cr); + format %{ "# cast_checked_LL $dst" %} + ins_encode %{ + __ verify_long_in_range(_idx, bottom_type()->is_long(), $dst$$Register, noreg); + %} + ins_pipe(pipe_slow); +%} + +instruct castLL_checked(rRegL dst, rRegL tmp, rFlagsReg cr) +%{ + predicate(VerifyConstraintCasts > 0 && !castLL_is_imm32(n)); + match(Set dst (CastLL dst)); + + effect(KILL cr, TEMP tmp); + format %{ "# cast_checked_LL $dst\tusing $tmp as TEMP" %} + ins_encode %{ + __ verify_long_in_range(_idx, bottom_type()->is_long(), $dst$$Register, $tmp$$Register); + %} + ins_pipe(pipe_slow); +%} + instruct castFF(regF dst) %{ match(Set dst (CastFF dst)); @@ -8538,7 +8566,6 @@ instruct mulI_rReg_ndd(rRegI dst, rRegI src1, rRegI src2, rFlagsReg cr) instruct mulI_rReg_imm(rRegI dst, rRegI src, immI imm, rFlagsReg cr) %{ - predicate(!UseAPX); match(Set dst (MulI src imm)); effect(KILL cr); @@ -8550,20 +8577,6 @@ instruct mulI_rReg_imm(rRegI dst, rRegI src, immI imm, rFlagsReg cr) ins_pipe(ialu_reg_reg_alu0); %} -instruct mulI_rReg_rReg_imm_ndd(rRegI dst, rRegI src1, immI src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (MulI src1 src2)); - effect(KILL cr); - - ins_cost(300); - format %{ "eimull $dst, $src1, $src2\t# int ndd" %} - ins_encode %{ - __ eimull($dst$$Register, $src1$$Register, $src2$$constant, false); - %} - ins_pipe(ialu_reg_reg_alu0); -%} - instruct mulI_mem(rRegI dst, memory src, rFlagsReg cr) %{ predicate(!UseAPX); @@ -8594,7 +8607,6 @@ instruct mulI_rReg_rReg_mem_ndd(rRegI dst, rRegI src1, memory src2, rFlagsReg cr instruct mulI_mem_imm(rRegI dst, memory src, immI imm, rFlagsReg cr) %{ - predicate(!UseAPX); match(Set dst (MulI (LoadI src) imm)); effect(KILL cr); @@ -8606,20 +8618,6 @@ instruct mulI_mem_imm(rRegI dst, memory src, immI imm, rFlagsReg cr) ins_pipe(ialu_reg_mem_alu0); %} -instruct mulI_rReg_mem_imm(rRegI dst, memory src1, immI src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (MulI (LoadI src1) src2)); - effect(KILL cr); - - ins_cost(300); - format %{ "eimull $dst, $src1, $src2\t# int ndd" %} - ins_encode %{ - __ eimull($dst$$Register, $src1$$Address, $src2$$constant, false); - %} - ins_pipe(ialu_reg_mem_alu0); -%} - instruct mulAddS2I_rReg(rRegI dst, rRegI src1, rRegI src2, rRegI src3, rFlagsReg cr) %{ match(Set dst (MulAddS2I (Binary dst src1) (Binary src2 src3))); @@ -8660,7 +8658,6 @@ instruct mulL_rReg_ndd(rRegL dst, rRegL src1, rRegL src2, rFlagsReg cr) instruct mulL_rReg_imm(rRegL dst, rRegL src, immL32 imm, rFlagsReg cr) %{ - predicate(!UseAPX); match(Set dst (MulL src imm)); effect(KILL cr); @@ -8672,20 +8669,6 @@ instruct mulL_rReg_imm(rRegL dst, rRegL src, immL32 imm, rFlagsReg cr) ins_pipe(ialu_reg_reg_alu0); %} -instruct mulL_rReg_rReg_imm_ndd(rRegL dst, rRegL src1, immL32 src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (MulL src1 src2)); - effect(KILL cr); - - ins_cost(300); - format %{ "eimulq $dst, $src1, $src2\t# long ndd" %} - ins_encode %{ - __ eimulq($dst$$Register, $src1$$Register, $src2$$constant, false); - %} - ins_pipe(ialu_reg_reg_alu0); -%} - instruct mulL_mem(rRegL dst, memory src, rFlagsReg cr) %{ predicate(!UseAPX); @@ -8716,7 +8699,6 @@ instruct mulL_rReg_rReg_mem_ndd(rRegL dst, rRegL src1, memory src2, rFlagsReg cr instruct mulL_mem_imm(rRegL dst, memory src, immL32 imm, rFlagsReg cr) %{ - predicate(!UseAPX); match(Set dst (MulL (LoadL src) imm)); effect(KILL cr); @@ -8728,20 +8710,6 @@ instruct mulL_mem_imm(rRegL dst, memory src, immL32 imm, rFlagsReg cr) ins_pipe(ialu_reg_mem_alu0); %} -instruct mulL_rReg_mem_imm_ndd(rRegL dst, memory src1, immL32 src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (MulL (LoadL src1) src2)); - effect(KILL cr); - - ins_cost(300); - format %{ "eimulq $dst, $src1, $src2\t# long ndd" %} - ins_encode %{ - __ eimulq($dst$$Register, $src1$$Address, $src2$$constant, false); - %} - ins_pipe(ialu_reg_mem_alu0); -%} - instruct mulHiL_rReg(rdx_RegL dst, rRegL src, rax_RegL rax, rFlagsReg cr) %{ match(Set dst (MulHiL src rax)); @@ -10631,21 +10599,6 @@ instruct xorI_rReg_rReg_mem_ndd(rRegI dst, rRegI src1, memory src2, rFlagsReg cr ins_pipe(ialu_reg_mem); %} -instruct xorI_rReg_mem_rReg_ndd(rRegI dst, memory src1, rRegI src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (XorI (LoadI src1) src2)); - effect(KILL cr); - flag(PD::Flag_sets_sign_flag, PD::Flag_sets_zero_flag, PD::Flag_sets_parity_flag, PD::Flag_clears_overflow_flag, PD::Flag_clears_carry_flag); - - ins_cost(150); - format %{ "exorl $dst, $src1, $src2\t# int ndd" %} - ins_encode %{ - __ exorl($dst$$Register, $src1$$Address, $src2$$Register, false); - %} - ins_pipe(ialu_reg_mem); -%} - // Xor Memory with Register instruct xorB_mem_rReg(memory dst, rRegI src, rFlagsReg cr) %{ @@ -10825,21 +10778,6 @@ instruct andL_rReg_rReg_mem_ndd(rRegL dst, rRegL src1, memory src2, rFlagsReg cr ins_pipe(ialu_reg_mem); %} -instruct andL_rReg_mem_rReg_ndd(rRegL dst, memory src1, rRegL src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (AndL (LoadL src1) src2)); - effect(KILL cr); - flag(PD::Flag_sets_sign_flag, PD::Flag_sets_zero_flag, PD::Flag_sets_parity_flag, PD::Flag_clears_overflow_flag, PD::Flag_clears_carry_flag); - - ins_cost(150); - format %{ "eandq $dst, $src1, $src2\t# long ndd" %} - ins_encode %{ - __ eandq($dst$$Register, $src1$$Address, $src2$$Register, false); - %} - ins_pipe(ialu_reg_mem); -%} - // And Memory with Register instruct andL_mem_rReg(memory dst, rRegL src, rFlagsReg cr) %{ @@ -11335,21 +11273,6 @@ instruct xorL_rReg_rReg_mem_ndd(rRegL dst, rRegL src1, memory src2, rFlagsReg cr ins_pipe(ialu_reg_mem); %} -instruct xorL_rReg_mem_rReg_ndd(rRegL dst, memory src1, rRegL src2, rFlagsReg cr) -%{ - predicate(UseAPX); - match(Set dst (XorL (LoadL src1) src2)); - effect(KILL cr); - flag(PD::Flag_sets_sign_flag, PD::Flag_sets_zero_flag, PD::Flag_sets_parity_flag, PD::Flag_clears_overflow_flag, PD::Flag_clears_carry_flag); - - ins_cost(150); - format %{ "exorq $dst, $src1, $src2\t# long ndd" %} - ins_encode %{ - __ exorq($dst$$Register, $src1$$Address, $src2$$Register, false); - %} - ins_pipe(ialu_reg_mem); -%} - // Xor Memory with Register instruct xorL_mem_rReg(memory dst, rRegL src, rFlagsReg cr) %{ diff --git a/src/hotspot/cpu/zero/icache_zero.hpp b/src/hotspot/cpu/zero/icache_zero.hpp index b40e07d5e3b..781021a2b20 100644 --- a/src/hotspot/cpu/zero/icache_zero.hpp +++ b/src/hotspot/cpu/zero/icache_zero.hpp @@ -33,7 +33,7 @@ class ICache : public AbstractICache { public: - static void initialize() {} + static void initialize(int phase) {} static void invalidate_word(address addr) {} static void invalidate_range(address start, int nbytes) {} }; diff --git a/src/hotspot/cpu/zero/sharedRuntime_zero.cpp b/src/hotspot/cpu/zero/sharedRuntime_zero.cpp index f141135ff95..60a873ab31f 100644 --- a/src/hotspot/cpu/zero/sharedRuntime_zero.cpp +++ b/src/hotspot/cpu/zero/sharedRuntime_zero.cpp @@ -50,18 +50,17 @@ int SharedRuntime::java_calling_convention(const BasicType *sig_bt, return 0; } -AdapterHandlerEntry* SharedRuntime::generate_i2c2i_adapters( - MacroAssembler *masm, - int total_args_passed, - int comp_args_on_stack, - const BasicType *sig_bt, - const VMRegPair *regs, - AdapterFingerPrint *fingerprint) { - return AdapterHandlerLibrary::new_entry( - fingerprint, - CAST_FROM_FN_PTR(address,zero_null_code_stub), - CAST_FROM_FN_PTR(address,zero_null_code_stub), - CAST_FROM_FN_PTR(address,zero_null_code_stub)); +void SharedRuntime::generate_i2c2i_adapters(MacroAssembler *masm, + int total_args_passed, + int comp_args_on_stack, + const BasicType *sig_bt, + const VMRegPair *regs, + AdapterHandlerEntry* handler) { + handler->set_entry_points(CAST_FROM_FN_PTR(address,zero_null_code_stub), + CAST_FROM_FN_PTR(address,zero_null_code_stub), + CAST_FROM_FN_PTR(address,zero_null_code_stub), + nullptr); + return; } nmethod *SharedRuntime::generate_native_wrapper(MacroAssembler *masm, diff --git a/src/hotspot/cpu/zero/vm_version_zero.cpp b/src/hotspot/cpu/zero/vm_version_zero.cpp index e38561e19c5..3ce9227c193 100644 --- a/src/hotspot/cpu/zero/vm_version_zero.cpp +++ b/src/hotspot/cpu/zero/vm_version_zero.cpp @@ -151,6 +151,6 @@ void VM_Version::initialize_cpu_information(void) { _no_of_threads = _no_of_cores; _no_of_sockets = _no_of_cores; snprintf(_cpu_name, CPU_TYPE_DESC_BUF_SIZE - 1, "Zero VM"); - snprintf(_cpu_desc, CPU_DETAILED_DESC_BUF_SIZE, "%s", _features_string); + snprintf(_cpu_desc, CPU_DETAILED_DESC_BUF_SIZE, "%s", _cpu_info_string); _initialized = true; } diff --git a/src/hotspot/os/aix/libodm_aix.cpp b/src/hotspot/os/aix/libodm_aix.cpp index 9fe0fb7abd8..854fd5e2b79 100644 --- a/src/hotspot/os/aix/libodm_aix.cpp +++ b/src/hotspot/os/aix/libodm_aix.cpp @@ -30,6 +30,7 @@ #include