The Issue

Developer reported to me that there is handshake problem with an internal API gateway.

 javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Troubleshooting

I found curl and wget works well. So I decided to do a packet capture with tcpdump:

sudo /usr/sbin/tcpdump -Z appadm -w api.pcap host 10.xxx.xxx.xxx

Here are facts:

1. curl can get Server Hello and go through the complete SSL handshake
2. Java cannot get Server Hello
3. Both are using TLS 1.2

So I doubt it is about the Cipher Suites. The Java supported Cipher Suites are:

Cipher Suites (29 suites)
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
    Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
    Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026)
    Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a)
    Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
    Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
    Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
    Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
    Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
    Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
    Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
    Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
    Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025)
    Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
    Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
    Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
    Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
    Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
    Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
    Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
    Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
    Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)

The curl pcap uses TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 which is not in the list. So I used following command to scan the supported cipher suites:

[appadm local]$ nmap --script ssl-enum-ciphers -p 443 lab-api-int.xxx.com

Starting Nmap 5.51 ( http://nmap.org ) at 2020-06-29 22:58 UTC
Nmap scan report for lab-api-int.xxx.com (10.xxx.xxx.xxx)
Host is up (0.00038s latency).
rDNS record for 10.yyy.yyy.yyy: lab2-api-int.yyy.yyy.com
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers: 
|   TLSv1.2
|     Ciphers (2)
|       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|     Compressors (1)
|_      uncompressed

Nmap done: 1 IP address (1 host up) scanned in 0.77 seconds

So it is clear the handshake failure is about the Cipher Suites. Per stackover flow (https://stackoverflow.com/questions/21289293/java-7-support-of-aes-gcm-in-ssl-tls), JDK 1.7, which is in use, doesn’t support AES-GCM cipher suites.

In this case, I had to ask developers to turn to JDK 1.8 and it works well!

My Requirement

I was trying to add some information regarding my code into my Flask app. Like version and build time. This is needed for all templates as a footer it should be processed once.

My version information is stored in __version__.py in this format:

__title__ = 'fcapp'
__description__ = 'My app'
__url__ = 'https://felixchr.github.com/'
__version__='1.0.4'
__build__='0x5ee41d6d'
__author__ = 'Felix Cao'
__author_email__ = 'felix.cao@XXX.com'
__license__ = 'MIT'

The Solution

I studied online to find out the resolution. And eventually I found it on stackoverflow and here is my code here:

__init__.py of my Flask app

I added code to read and load the variables immediately after app is initialized:

app = Flask(__name__)
app.config.from_object('config')

@app.context_processor
def inject_stage_and_region():
    here = os.path.abspath(os.path.dirname(__file__))
    about = dict()
    with open(os.path.join(here, '__version__.py'), 'r') as f:
        exec(f.read(), about)
return {'about': about}

The return value is a dict. Part of this code is borrowed from Kenneth Reitz’’s requests.

The base.html of my main template

In my base.html now I can add those to this code:

  <div>
      <hr/>
      Version: {{about.__version__}}<br/>
      Build Time: {{about.__build__}}
  </div>

It works now!

The Background

I need to develop Python and Bash scripts for our applications which running on Linux but my workstation is a Windows 10 laptop. I would like to test immediately on my local machine instead of test on remote servers, in spite of that I have Jenkins set up for testing. My company’s VPN has very tough network restrictions. It force all outbound traffic goes to VPN gateway. That is to say, I cannot reach to any other IP addresses other than 127.0.0.1 locally.

Install and Configure VirtualBox

I would not elaborate how to install VirtualBox on Windows 10. It is straightforward. Just download and install.

Guest OS installation is not my topic as well.

Network Configuration

Network Types

Bridge: act like another computer in the same network. It can get same IP addresses with your computer.

NAT: Use your computer as gateway. Both your computer and VM will have IP in same subnet

Host-only: accessible only from Host (I’m not if this is correct. Maybe updated later)

Use NAT Network

Due to the restriction of my company’s VPN, I had to use NAT Network for my guest OS.

In Network tab of Settings of VM, set the adapter to attach to NAT Network.

The default network is 10.0.2.0/24. 10.0.2.1 and 10.0.2.2 are occupied. You’d better to use static instead of DHCP because we’ll need to set Port Forwarding. You won’t want to change the Port Forwarding frequently if you have multiple VMs.

Port Forwarding

Port Forwarding is globally. It’s in File -> Preference -> Network -> NatNetwork. Click Edit NatNetwork and Port Forwarding.

Share Folder Between Host and Guest

Use Samba

This was tested with VMWare Player and CentOS 6. After I switched to VirtualBox, I didn’t try it as my VPN restricts me to access any IP other than 127.0.0.1. I put it here as this may help your situation.

1. Install samba

$ sudo yum -y install samba

1. Configure Samba server

   The configuration file is /etc/samba/smb.conf

[global]
    workgroup = MYGROUP
    server string = Samba Server Version %v
    log file = /var/log/samba/log.%m
    max log size = 50
    security = user
    passdb backend = tdbsam
    load printers = no
    cups options = raw
[homes]
    comment = Home Directories
    browseable = no
    writable = yes

1. configure password

[root@diamond samba]# smbpasswd -a felixc
New SMB password:
Retype new SMB password:
Added user felixc.

1. Mount on Windows

   On windows, run: \<IP-ADDR>Use “local\usernmae” and password to login.

Use VirtualBox shared folder

This is tested with VirtualBox 6.1 and CentOS 7. This is my current test environment.

1. install guest addons 1.1 Devices -> Insert Guest Additions CD images

   1.2 update kernel on Guest OS

$ sudo yum update kernel -y
$ sudo yum install kernel-headers kernel-devel gcc make -y
$ sudo init 6

1.3. mount cdrom and install the additions to OS.

# check the cdrom device
$ blkid
$ sudo mount /dev/sr0 /a
$ cd /a
$ sudo ./VBoxLinuxAdditions.run

1. set up share folder in VirtualBox on host

   Use menu VM -> Settings -> Shared Folders,

2. Add user to vboxfs group in order to gain permission to the shared folder

$ sudo usermod -G vboxfs felixc

1. mount on guest OS

$ VBoxControl sharedfolder list
$ mount.vboxsf share /app

Development Environment

Use VS Code Remote Development Plugin

VS Code is awesome! The Remote Development plugin works good with Linux environment.

I installed following plugins on my host for remote development:

1. Remote Development
2. Remote - SSH
3. Remote - SSH: Editing Configuration Files
4. Remote - WSL

Following plugins are installed on develop server:

1. Python
2. GitLens
3. Prettier

VS Code supports pipenv very well. It loads python virtualenv automatically.

Use Git Bash for Windows

Git bash is a very good way to provide Bash environment and GIT. It would be best alternative for Windows Command Line.

Issues and Fixes

1. rc=-5640

This is due to some antivirus software. So far I haven’t found any fix for it. But you can try to few times to start your VM. Sometimes I tried more than ten times in order to start my VM.

Someone says increasing the memory size would increase the starting success rate. I set my VM to 2GB RAM and it seems better now.

1. Cannot PING gateway under NAT Network

This happens very often. To fix it, use VBoxManage to restart the network. I wrote a bash script and put it into PATH of my Git Bash. Each time when I have this issue I run it.

/c/fc/local/bin/vboxnat-restart.sh

#!/bin/bash
echo "Stopping..."
/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe natnetwork stop --netname NatNetwork
sleep 2
echo "Starting..."
/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe natnetwork start --netname NatNetwork
echo "Done"