Almost a decade ago, Medium was a nice platform, but it has changed over the years. How? They monetized everything.

If you don’t care, they will still do it for you.

This blog migrated a while ago to https://kirill.korins.ky/

Enjoy!

For example, macOS 10.14 can’t connect to anything which is used Let’s Encrypt SSL.

The good news is that it is quite easy to fix. Just run:

This simple code is removing an expired certificate.

The root cause of this issue is a bug inside LibreSSL which was used in macOS 10.14 and cross-self-signing between roots Let’s Encrypt’s root CAs.

I assume that you’re using MacPorts that is working :)

First, you need something that we call bootstrap java. AdoptOpenJDK 1.8 isn’t ported and MacPorts complains about it, but we need it.

So, let install it and SBT.

catap@Kirills-mini-m1 ~ % sudo port install openjdk8 sbt  
--->  Installing openjdk8 @8u275_0
--->  Activating openjdk8 @8u275_0
--->  Cleaning openjdk8
--->  Computing dependencies for sbt
--->  Fetching archive for sbt
--->  Attempting to fetch sbt-0.13.18_0.darwin_20.noarch.tbz2 from https://packages.macports.org/sbt
--->  Attempting to fetch sbt-0.13.18_0.darwin_20.noarch.tbz2 from https://lil.fr.packages.macports.org/sbt
--->  Attempting to fetch sbt-0.13.18_0.darwin_20.noarch.tbz2 from https://nue.de.packages.macports.org/sbt
--->  Fetching distfiles for sbt
Warning: Failed to confirm that required Java was installed; see https://trac.macports.org/ticket/61445
--->  Verifying checksums for sbt
--->  Extracting sbt
--->  Configuring sbt
--->  Building sbt
--->  Staging sbt into destroot
--->  Installing sbt @0.13.18_0
--->  Activating sbt @0.13.18_0
--->  Cleaning sbt
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  Found 8 broken files, matching files to ports
--->  Found 1 broken port, determining rebuild order
You can always run 'port rev-upgrade' again to fix errors.
The following ports will be rebuilt: openjdk8 @8u275
Continue? [Y/n]: n
--->  Some of the ports you installed have notes:
  openjdk8 has the following notes:
    If you have more than one JDK installed you can make openjdk8 the default
    by adding the following line to your shell profile:
    
        export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk8/Contents/Home
catap@Kirills-mini-m1 ~ %

Good, we have very old SBT-0.13.8 and JDK that isn’t native and ports complaints to it. Let start to fix it.

Now we need a brand, new, unreleased (yet) SBT that we will use as boot sbt. Let build one. To do that we need unreleased upstream from SBT and one patch to switch some dependencies to the latest version that contains fixes.

catap@Kirills-mini-m1 ~ % mkdir -p ~/src
catap@Kirills-mini-m1 ~ % cd ~/src
catap@Kirills-mini-m1 src % git clone https://github.com/sbt/sbt.git                                                            
Cloning into 'sbt'...
remote: Enumerating objects: 238, done.
remote: Counting objects: 100% (238/238), done.
remote: Compressing objects: 100% (169/169), done.
remote: Total 112828 (delta 81), reused 147 (delta 44), pack-reused 112590
Receiving objects: 100% (112828/112828), 25.58 MiB | 14.42 MiB/s, done.
Resolving deltas: 100% (52326/52326), done.
catap@Kirills-mini-m1 src % cd sbt
catap@Kirills-mini-m1 sbt % curl -s https://gist.githubusercontent.com/catap/8396d51fbb0242e800aa05e395f16366/raw/656c532757f710e07ad177a4154f59968d0c780d/sbt-dependencies.patch | patch -p1
patching file build.sbt
patching file project/Dependencies.scala
patch unexpectedly ends in middle of line
Hunk #1 succeeded at 26 with fuzz 1.
catap@Kirills-mini-m1 sbt % sbt publishLocal > /dev/null     
catap@Kirills-mini-m1 sbt %

Good, now we should flush boot cache of sbt:

catap@Kirills-mini-m1 sbt % rm -rf ~/.sbt/boot
catap@Kirills-mini-m1 sbt %

force SBT to use our version:

catap@Kirills-mini-m1 sbt % cat > ~/.sbtconfig
JAVA_OPTS="-Dsbt.version=1.5.0-SNAPSHOT"
^D
catap@Kirills-mini-m1 sbt %

and switch IPC inside SBT to TCP

catap@Kirills-mini-m1 sbt % cat > ~/.sbt/1.0/global.sbt 
Global / serverConnectionType := ConnectionType.Tcp
^D
catap@Kirills-mini-m1 sbt %

Good. Now we can remove AdoptOpenJDK and install ZuluJDK8 or ZuluJDK11 that is native for this platform:

catap@Kirills-mini-m1 sbt % sudo port uninstall openjdk8
Note: It is not recommended to uninstall/deactivate a port that has dependents as it breaks the dependents.
The following ports will break:
 sbt @0.13.18_0
 sbt @0.13.18_0
Continue? [y/N]: y
Warning: Uninstall forced.  Proceeding despite dependencies.
--->  Deactivating openjdk8 @8u275_0
--->  Cleaning openjdk8
--->  Uninstalling openjdk8 @8u275_0
--->  Cleaning openjdk8
catap@Kirills-mini-m1 sbt % sudo port install zulu-jdk11
--->  Computing dependencies for zulu-jdk11
--->  Fetching distfiles for zulu-jdk11
--->  Verifying checksums for zulu-jdk11
--->  Extracting zulu-jdk11
--->  Configuring zulu-jdk11
--->  Building zulu-jdk11
--->  Staging zulu-jdk11 into destroot
Warning: zulu-jdk11 installs files outside the common directory structure.
--->  Installing zulu-jdk11 @11.43.1007_0
--->  Activating zulu-jdk11 @11.43.1007_0
--->  Cleaning zulu-jdk11
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.
--->  Some of the ports you installed have notes:
  zulu-jdk11 has the following notes:
    If you have more than one JDK installed you can make zulu-jdk11 the default
    by adding the following line to your shell profile:
    
        export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-jdk11/Contents/Home
catap@Kirills-mini-m1 sbt %

Congrats! You have SBT that is working fine on Apple Silicon.

catap@Kirills-mini-m1 sbt % mkdir -p /tmp/sbt-test 
catap@Kirills-mini-m1 sbt % cd /tmp/sbt-test 
catap@Kirills-mini-m1 sbt-test % sbt
[info] Updated file /private/tmp/sbt-test/project/build.properties: set sbt.version to 1.5.0-SNAPSHOT
[info] welcome to sbt 1.5.0-SNAPSHOT (Azul Systems, Inc. Java 11.0.9.1)
[info] loading project definition from /private/tmp/sbt-test/project
[info] set current project to sbt-test (in build file:/private/tmp/sbt-test/)
[info] sbt server started at tcp://127.0.0.1:5715
[info] started sbt server
sbt:sbt-test>

I hope soon everything will be merged, and you will need just upgrade SBT.

P.S. SBT 1.4.3 supports Apple Silicon. You can use 1.4.x branch and do not apply any patches.

Why?

The cause is openjdk/jdk@113c48f that introduced isEmpty method to java.lang.CharSequence since Java 15.

My code can’t be compiled on Java 15+ because isEmpty can be used from scala.collection.ArrayOps[Char] and from ArrayCharSequence that inherits java.lang.CharSequence with a brand new isEmpty.

Can it be solve? Only with hack at Scala compiler.

The good news that anything that was build before can be run on Java 15.

Scala usessize method inside ArrayOps to overstep similar issue with length that is implemented inside Java’s array for example and I have no idea how it is possibly to solve without renaming method on Scala side.

Anyway you can enjoy discussion.

We had a bit data from WHO here: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports/

We can extract from it that number of cases was:

44 cases at 3rd day of the year

282 cases at 20th day of the year

314 cases at 21st day of the year

581 cases at 22nd day of the year

846 cases at 24th day of the year

1320 cases at 25th day of the year

2014 [was predicted as 1843, error 9.2%] as 26th day of the year

2798 [was predicted as 2950, error 5.1%] as 27th day of the year

It looks like exponential regression, isn’t it?

Let use WolframAlpha to draw it: https://www.wolframalpha.com/input/?i=exponential+fit+%7B%283%2C44%29%2C%2820%2C282%29%2C%2821%2C314%29%2C%2823%2C581%29%2C%2824%2C846%29%2C%2825%2C1320%29%2C%2826%2C2014%29%7D

Well, we have a formula: 0.0671343 e^(0.395955 x)

Let replace x to some days in the near future:

1st February (32nd day): 21,364 cases

14th February (45th day): 3.7 millions cases

1st March (60th day): 1.4 billions cases

5th March (64th day): about 7 billions cases

Congrats, everyone is infected!

This experiment was made on unnamed cluster with dozens of machine and I’d like to share only two pictures. One from node with enabled Shenandoah for last week, and second one has G1 with settings closer to that I described before. Both machine has very similar number of segments and this cluster uses almost for writing. Let say 99% of operation it is writing.

Summary: at peaks loading it is much better and at night it is similar. So, it is expected behaviour to be honest. Also it removes full GC that brokes this cluster time-to-time.

You can see some result at the graph for last 24 hours:

But it has two issues:

1. Internal metrics for GC counts doesn’t work at all.
2. Shenandoah makes a lot of things as separated non blocking threads and it brokes logic of JvmGcMonitorService, and logs full of lines like:

[2018-12-26T12:57:40,974][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422543] overhead, spent [770ms] collecting in the last [1s]

[2018-12-26T12:57:50,976][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422553] overhead, spent [1.3s] collecting in the last [1s]

[2018-12-26T12:57:58,978][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422561] overhead, spent [929ms] collecting in the last [1s]

[2018-12-26T12:58:06,979][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422569] overhead, spent [946ms] collecting in the last [1s]

[2018-12-26T12:58:15,981][WARN ][o.e.m.j.JvmGcMonitorService] [es9-main] [gc][422578] overhead, spent [749ms] collecting in the last [1s]

Costs? Well, it increases CPU usage but it is expected behaviour.

You may have different root causes, for example:

• JVM was killed by OOM-killer for some reason;
• hardware issue when you lose something on your disk;
• power failure;
• elastic bugs...

Anyway, it may happen. And it will happen.

Good news is elastic very tolerant of this sort of issue and may self-fix. But it can’t if you haven’t got a replica of this shard.

You may increase the probability of this by setup index.translog.durability to async but in some cases it may increase write speed. I don’t think that you really need it, but if you need… ;)

Anyway, one day you will have red status and if you run GET _cluster/allocation/explain you can find that one of shard has an issue and the system hasn’t got another copy of this shard…

OK, that should you do?

You can run something likes:

after that, you will recreate this shard. Yes, you will have the green status again but lose your data. You can use allocate_stale_primary if some nodes leave cluster and it may contain missed shard.

But if you haven’t got a node that left the cluster and you had only one copy of shard and it was corrupted… you still have an option before lose your data.

At the explain you can find a path to the corrupted Lucene index. Something likes SimpleFSDirectory@/var/lib/elasticsearch/some-node/nodes/0/indices/BYurcTitskeSexyYOBjiu1/66/index

If this folder contains a bit more than a few files a dozen bytes each you still have a chance. How?

Try to run:

sudo java -cp /usr/share/elasticsearch/lib/elasticsearch-*.jar:/usr/share/elasticsearch/lib/* org.apache.lucene.index.CheckIndex [path]

If it found any errors you can try to fix it. How? First of all shutdown the node on the machine. You can try to fix it on the fly, but ask elastic to reread corrupted index is really tricky and unsafe, so, I suggest to don’t do it and just stop it.

After you’ve stopped it re-run the same command but add -exorcise option after the path.

If it can fix it, it will. And if it fixed something you may start your node… and have a green status.

If it can’t? Well… You still may write some code to try to fix your specified case… But I suggest to copy broken index to some safe place, start everything back and remove this shard. Because you have a copy of the index, you can extract all survivors part and re-index it by hand.

If you think that your data is very important and you have no idea how to try to get it… You still can drop me an email and I can try to help ;)