Skip to content

Feature: Mutual TLS (mTLS) support with client certificate authentication#62

Open
crazyrokr wants to merge 7 commits intoJavaSaBr:developfrom
crazyrokr:mtls-support
Open

Feature: Mutual TLS (mTLS) support with client certificate authentication#62
crazyrokr wants to merge 7 commits intoJavaSaBr:developfrom
crazyrokr:mtls-support

Conversation

@crazyrokr
Copy link
Copy Markdown
Contributor

@crazyrokr crazyrokr commented Apr 21, 2026

Adds StringDataMtlsServerConnection that enforces client certificate authentication, along with tests and fixes for the SSL handshake state machine.

Changes

  1. StringDataMtlsServerConnection - new connection type that calls sslEngine.setNeedClientAuth(true) before the handshake starts.
  2. AbstractSslConnection - removed beginHandshake() from the constructor and exposed it as a public method, so subclasses can configure the SSLEngine (e.g. setNeedClientAuth) before the handshake begins.
  3. NetworkFactory - added stringDataMtlsServerNetwork factory method; all SSL connections now call connection.beginHandshake() explicitly after construction.
  4. AbstractSslNetworkPacketWriter.doHandshake() - fixed an infinite loop: the NEED_UNWRAP branch had break instead of return EMPTY_BUFFER, which blocked the single-threaded network executor during the mTLS handshake.
  5. rlib_test_client_cert.p12 - added test client certificate (PKCS12, password testpw), used as both the client keystore and the server-side trust store.
  6. Fixed a data loss issue described below.

Problem

The issue was identified in AbstractSslNetworkPacketReader during the SSL/TLS handshake process. This occurs when a single network packet contains both the final handshake data and the initial encrypted application data.

Previously, if sslEngine.unwrap() resulted in a NEED_WRAP status (indicating the engine needs to send a handshake response), the reader would immediately request a wrap and then clear the remaining data in the buffer. This caused any application data that arrived in the same packet to be discarded, leading to protocol desynchronization or connection hangs.

Fix

In AbstractSslNetworkPacketReader.doHandshake(), a check was added within the NEED_WRAP case. Before cleaning the buffer and returning, the reader now checks if networkBuffer still has remaining bytes. If it does, these bytes are immediately passed to decryptAndRead() to be decrypted and processed as application packets.

Comment thread rlib-network/src/main/java/javasabr/rlib/network/impl/AbstractSslConnection.java Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds mutual TLS (mTLS) support to rlib-network by introducing an mTLS-capable server connection type, updating SSL connection initialization to allow SSLEngine configuration before the handshake, and fixing an SSL handshake writer state issue that could stall the network executor.

Changes:

  • Introduces StringDataMtlsServerConnection to require client certificate authentication (setNeedClientAuth(true)).
  • Refactors SSL connection initialization so handshakes are started explicitly via beginHandshake() after construction.
  • Fixes AbstractSslNetworkPacketWriter.doHandshake() behavior for NEED_UNWRAP to avoid executor stalls; adds integration tests and a client certificate keystore.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
rlib-network/src/test/resources/ssl/rlib_test_client_cert.p12 Adds a PKCS12 client cert keystore used by mTLS integration tests.
rlib-network/src/test/java/javasabr/rlib/network/SslMutualTlsTest.java Adds integration/regression tests covering mTLS acceptance/rejection scenarios.
rlib-network/src/main/java/javasabr/rlib/network/packet/impl/AbstractSslNetworkPacketWriter.java Fixes handshake handling for NEED_UNWRAP to prevent stalling.
rlib-network/src/main/java/javasabr/rlib/network/impl/StringDataMtlsServerConnection.java New server connection type that enforces client cert authentication.
rlib-network/src/main/java/javasabr/rlib/network/impl/AbstractSslConnection.java Moves beginHandshake() out of the constructor into an explicit method.
rlib-network/src/main/java/javasabr/rlib/network/NetworkFactory.java Updates SSL connection factories to call beginHandshake() explicitly; adds mTLS server factory method.

Comment thread rlib-network/src/main/java/javasabr/rlib/network/impl/AbstractSslConnection.java Outdated
Comment thread rlib-network/src/test/java/javasabr/rlib/network/SslMutualTlsTest.java Outdated
Comment thread rlib-network/src/test/java/javasabr/rlib/network/SslMutualTlsTest.java Outdated
crazyrokr and others added 4 commits April 22, 2026 13:44
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants